@dsh-external/dsh-vision-toolkit

Configuration fields

Field	Default	Contract
provider.baseUrl	https://api.inferera.com/v1	OpenAI-compatible base URL; normalized without trailing slashes
provider.credential	VISION_API_KEY	DSH Credential reference, never a secret value
provider.model	gemini-3.6-flash	Multimodal model name sent to remote tools
language	zh	Vision output language: zh or en
timeoutMs	60000	Whole-operation deadline, 1000-600000 ms; each tool may request a narrower override
maxImageBytes	10485760	Encoded-byte limit per input image
maxImagePixels	40000000	Decoded-pixel limit per input image
concurrency	4	In-flight operations per session, 1-16
runtime.mode	managed	managed uses the packaged snapshot; external accepts only the exact pin
runtime.agentVisionToolkitPath	unset	Required in external mode; exported exact snapshot or clean pinned Git checkout
runtime.python	unset	Optional Python 3.11+ bootstrap/interpreter override
allowedDirs	[]	Additional realpath-resolved input roots; the session workspace is always allowed

Usage patterns

Basic calls

vision_glance images=["screenshot.png"] query="What error is shown?"
vision_ground image="screenshot.png" target="the send button" preview=true
vision_detect image="screenshot.png" category="buttons" preview=true
vision_crop image="screenshot.png" region="1067,841,1108,881"
vision_trace image="icon.png" color=true output="icon.svg"
vision_pixel_diff original="reference.png" rebuilt="actual.png" runName="comparison"
vision_long_screenshot_ocr image="page.png" mode="general" jobs=2
vision_extract_foreground image="logo.png" mode="color"
vision_dominant_colors image="screen.png" region="0,0,600,300" top=8
vision_html_screenshot source="implementation.html" width=1200 height=720

Common workflows are vision_ground → vision_crop → vision_glance, vision_ground → vision_crop → vision_trace, and reference image → vision_html_screenshot → vision_pixel_diff. Grounding and detection boxes always use original-image pixels (x1/y1/x2/y2).

UI restoration example

The checked-in UI restoration example renders a reference, an intentionally inaccurate first implementation, and the final implementation through vision_html_screenshot, then compares both candidates through vision_pixel_diff:

npm run example:ui-restoration
npm run example:ui-restoration:write

The committed evidence records an initial 6.04% difference across six non-zero worst regions and a final 0% difference with no non-zero worst region. Check mode reproduces the tool path and verifies the committed assets; write mode intentionally refreshes the evidence.