@zseven-w/dsh-ios

DSH iOS Simulator

A live, interactive iOS Simulator inside a DeepSeek Harness conversation — plus your real iPhone over USB.

22 agent tools • live MJPEG sidebar panel • simulator & real iPhone over USB • list/feed row actions • SwiftUI preview hot reload

npm: @zseven-w/dsh-ios · Current plugin release: 0.1.0-rc.2 · Tested with DSH 0.1.0-rc.6

English · 简体中文 · 繁體中文 · 日本語 · 한국어 · Français · Español · Deutsch · Português · Русский · हिन्दी · Türkçe · ไทย · Tiếng Việt · Bahasa Indonesia

npm: @zseven-w/dsh-ios · Current plugin release: 0.1.0-rc.2 · Tested with DSH 0.1.0-rc.6

A real iPhone driven from inside a DSH conversation — the agent's tool calls on the left, the live device panel on the right

Why DSH iOS Simulator

DSH iOS Simulator gives the agent a real iOS Simulator inside the conversation — and gives you the pixels. The agent can boot a device, build and run an Xcode project or Swift package, drive the UI by accessibility identity or by OCR text, read unified logs, and inspect processes, backtraces, and leaks, while a live stream of the device renders in a persistent sidebar panel where you can tap, drag, rotate, and press Home directly on the video. The same verbs also work on a real iPhone connected over USB: the plugin builds and launches WebDriverAgent on the phone, tunnels its control and screen ports over loopback, and streams the device into the same panel, cards, and tools. No image blocks, no screen-recording files: visual bytes reach the UI only through signed, expiring URLs served by the DSH webserver.

🖥️ Live simulator in the conversation	A serve-sim MJPEG stream of the booted device, proxied through signed /_dsh/dsh-ios/* routes into a persistent right-side panel — the browser never touches serve-sim's port.
📱 Real iPhone over USB	ios_real_start_wda builds and launches WebDriverAgent on a connected phone and tunnels its control (REST) and screen (MJPEG) ports over loopback; the same panel, tools, cards, and status capsule then drive the phone. The device must be unlocked, and every real-account tap is gated by the plugin's identify-before-tap rules.
🛠️ 22 agent tools	Devices, boot/shutdown, screenshot, interact, build & run, unified logs, AXe-backed UI tree + tap-by-element, list/feed row actions, Vision OCR find/tap, SwiftUI preview hot reload, processes, backtrace, leaks, app info.
👆 Interactive panel	Tap and drag on the live video; Home / rotate / screenshot / refresh icon toolbar with hover tooltips; size modes (适应 · 50–125% · S/M/L); frame styles (无框 / 边框 / 真机框); drag-resize up to 960 px with double-click reset; landscape auto-widen.
🧾 List & feed rows	ios_sim_ui_rows turns deep accessibility snapshots into indexed rows with labels and generically parsed counters; ios_sim_tap_row taps inside a row at relative coordinates and verifies the action by the counter's expected ±1 change — the only reliable confirmation a list app offers.
🔐 Loopback-only transport	serve-sim binds 127.0.0.1 in a dedicated port range; every route requires a loopback peer, a loopback Host, and Fetch-Metadata/Origin checks; HMAC capabilities expire within 10 minutes. The WebDriverAgent control/MJPEG tunnels on a real device are loopback usbmux forwards under the same fence.
⚡ SwiftUI preview hot reload	ios_sim_preview generates a disposable host app outside your package, builds your previews as a dylib, and hot-swaps edits into the running simulator without relaunching (~2–5 s).
🧭 Semantic UI automation	ios_sim_ui_tree dumps the accessibility tree (AXe-backed) and ios_sim_tap_element taps by label or identifier; ios_sim_find_text OCRs the screen when the tree is empty or degenerate, and ios_sim_tap_text taps the matched text — identity- and text-based taps instead of guessed coordinates.

Tools

All 22 tools are registered on every host and return plain JSON — visual bytes reach the UI only through presentationMeta + signed routes, never as image blocks. Simulator udids route through simctl/serve-sim; physical-device udids route through WebDriverAgent automatically. On non-macOS hosts (or when serve-sim is unresolvable) the tools stay registered but fail with an explanatory error; the one exception is ios_sim_preview status, which truthfully reports { running: false } on any host.

Core simulator tools

Tool	What it does	Key parameters
ios_sim_devices	List the iOS Simulator devices available on this Mac (udid, name, runtime, state) and which are booted, plus any USB-connected physical iPhones under realDevices (udid, name, osVersion, model, state, developerMode). Use it to discover the udid or name to pass to the other tools.	—
ios_sim_boot	Boot a device and start its live serve-sim stream; the stream stays alive for the conversation so the panel can show the simulator live.	udid (required — udid or device name)
ios_sim_shutdown	Shut a device down; stops the stream when it targets that device.	udid (required)
ios_sim_screenshot	Capture a PNG and return a small JSON summary (path, bytes, dimensions, device); the image renders in the card/panel, never as an image block. Works on the streamed simulator and on a USB-connected phone via WebDriverAgent.	udid (optional — streamed device, else first booted)
ios_sim_interact	Interact with the streamed device — simulator or USB-connected phone: tap at normalized 0..1 coordinates, type text (US keyboard on a simulator), press a hardware button (home, lock, volumeUp…), scroll, or send a touch gesture; after the action settles (~300 ms) a fresh screenshot shows the effect.	action (required — tap/type/button/gesture/scroll), x/y, text, name, json
ios_sim_list_apps	List the apps INSTALLED on a booted simulator or a connected phone (bundle id, display name, version, system flag) — a third-party bundle id cannot be guessed, so list it or pass name to ios_sim_launch_app. A FAILED listing throws (e.g. "the device is not reachable by CoreDevice") instead of returning an empty list, so count: 0 always means the device really has no matching app.	udid (optional), query (case-insensitive substring over display name AND bundle id, CJK included), include_system (default false)
ios_sim_launch_app	Launch an installed app on a booted simulator or a connected phone — by bundleId, or by name (a case-insensitive display-name substring resolved through the same listing, CJK included). Exactly one of the two; a launch failure and an ambiguous name both come back with what to do next (ios_sim_build_run is for building one from source).	bundleId or name (exactly one), udid, relaunch
ios_sim_build_run	Build an .xcodeproj, .xcworkspace, or Swift package for the simulator, install the built .app, and launch it; pass a physical-device udid to build, install, and launch on the phone instead (requires Apple Development signing). On failure the result carries the filtered xcodebuild error tail. Takes minutes for a full build.	projectPath (required), scheme, udid (streamed → booted → newest-runtime iPhone, which is booted), configuration (default Debug)
ios_real_start_wda	Start WebDriverAgent (WDA) on a USB-connected physical iPhone — real devices only, never a simulator. Adopts an already-running WDA when one answers; otherwise runs the xcodebuild build/launch (a cold build takes minutes), then waits until WDA reports ready and returns the control/MJPEG ports the live panel streams through. Run this first when ios_sim_screenshot / ios_sim_interact / ios_sim_ui_tree / ios_sim_tap_element report WDA is not running for the device.	udid (required — physical-device udid from ios_sim_devices.realDevices)

UI-tree tools (AXe-backed)

Tool	What it does	Key parameters
ios_sim_ui_tree	Dump the frontmost app's accessibility element tree (labels, identifiers, values, frames in device points) plus the screen size in points — AXe on a simulator, WebDriverAgent on a USB-connected phone (depth-capped by default there: an uncapped snapshot of a busy app measures ~32 s / 751 KB, capped ~2 s); output is capped at ~40 KB (deepest levels pruned, truncated + hint set).	udid (optional), max_depth, filter (case-insensitive substring over label/identifier/type)
ios_sim_tap_element	Tap an element by identity — exact match first, then case-insensitive substring over identifier/label; nested duplicates collapse to one target, ambiguous matches list every candidate. The tap lands on the element center (AXe HID on a simulator, WebDriverAgent on a phone), then a ~300 ms screenshot shows the effect; pass expect_text / expect_gone and the tap plus its verification become one round trip (expected.matched).	udid (optional), identifier, label, expect_text, expect_gone

List & feed rows

List/feed apps aggregate each item into one accessibility cell whose label carries the whole summary and all its counters ("57 回复。18 喜欢。592 次查看") — there are no per-control child buttons to match, and the row cells only surface at a deep snapshot. These two tools expose that structure as rows and act inside a row.

Tool	What it does	Key parameters
ios_sim_ui_rows	Read the visible list/feed rows of the frontmost app as rows instead of a raw tree: each row reports its index, frame in points, the aggregated label, and the counters parsed out of that label (number + classifier token, e.g. 57 回复 → 回复=57, 中文 or English — no app vocabulary hardcoded). Rows only surface at a deep snapshot: on a phone the default max_depth is 60, costing ~15–25 s / ~0.5 MB per call (WDA serves requests serially) — keep the cheap observers (ios_sim_find_text / ios_sim_ui_tree) first. Counters are parsed heuristically and keys round-trip: pass a key exactly as listed to ios_sim_tap_row.expect_count. When no rows are found the result says why (depth too shallow / not a list screen / genuinely no accessibility information after a deep read) — a shallow read is never reported as "the app has no accessibility information"; off-screen rows are excluded and counted as omittedOffscreen.	udid (optional), max_depth (phone-only; default 60)
ios_sim_tap_row	Tap at a relative position inside one visible list row (reported by ios_sim_ui_rows: 0-based index; x/y as fractions of that row's frame — 0 = left/top edge, 1 = right/bottom, default 0.5 = center) on a simulator (AXe) or a USB-connected phone (WebDriverAgent). The row frame comes from a FRESH tree read, so no absolute screen coordinates are guessed; an out-of-range index FAILS (never clamps). Safety gate: with expect_count={key,delta} the tool verifies the action by re-reading the row label and checking the counter moved exactly +1/−1 (countCheck.verified); if the key is not among the row's parsed counters the tap is REFUSED before it happens — a real-device tap is never a probe. Without expect_count the tap still happens (an explicit row-relative position IS the identification) but nothing is verified.	udid (optional), index (required), x, y (fractions 0..1), max_depth, expect_count ({key, delta})

OCR tools (Vision)

Tool	What it does	Key parameters
ios_sim_find_text	OCR the CURRENT screen of a booted simulator or a USB-connected phone with the plugin-compiled Vision helper (accurate recognition, zh-Hans + en-US, compiled with swiftc on first use into ~/Library/Caches/dsh-ios/bin/ocr). Use it when the accessibility tree is empty or degenerate, for text rendered as graphics (badge counts, prices baked into images), or to independently verify what is on screen. Captures a fresh screenshot and returns {device, size, items:[{text, confidence, rect}]} — rects are device-point boxes (origin top-left), confidence-sorted, capped at ~40 KB (truncated drops the lowest-confidence tail; narrow with query or raise min_confidence).	udid (optional), query (case-insensitive substring), min_confidence (default 0.3)
ios_sim_tap_text	OCR the CURRENT screen and tap the center of the best text match — the same exact → case-insensitive-contains → candidate-list ambiguity rules as ios_sim_tap_element, for text the accessibility tree cannot see (no-a11y apps, badge counts, text baked into images). On a phone the tap lands at absolute device points through WebDriverAgent; on the streamed simulator it is sent normalized through the serve-sim control (run ios_sim_boot first). After ~300 ms a fresh screenshot shows the effect; pass expect_text / expect_gone and the tap plus its verification become one round trip (expected.matched). On a REAL device every tap has real consequences — never tap an unidentified control to find out what it does.	udid (optional), query (required), min_confidence, expect_text, expect_gone
ios_sim_wait_for	Wait until text appears or disappears on the screen, polling the same capture+OCR pipeline as ios_sim_find_text until the condition holds or the timeout expires (default 8 s, max 60 s). A timeout is a normal matched:false answer, never an error — one call instead of a find_text loop that costs ~1.2 s per round trip on a phone. On a match, item carries the OCR text, confidence, and rect in device points.	udid (optional), text (required), mode (appear/disappear), timeout_ms, min_confidence

Logs tool

Tool	What it does	Key parameters
ios_sim_logs	Read what a simulator app prints, from the device unified log: snapshot (log show --last , default 2m) or follow (bounded live capture for duration_seconds, default 10, max 60 — never a hanging stream). Output is capped at ~300 lines / 30 KB with a narrowing hint.	udid (optional), mode (snapshot/follow), duration, duration_seconds, bundle_id, predicate (raw NSPredicate, overrides bundle_id), level (default/info/debug), grep

Preview tool

Tool	What it does	Key parameters
ios_sim_preview	SwiftUI preview hot reload, live in the simulator: start (default) validates the package, generates a disposable host app in the plugin cache (never inside your package), builds the package as a dylib for the simulator, installs + launches the host, and watches the sources — every edit rebuilds and hot-swaps without relaunching (~2–5 s). Compiler errors keep the last good preview and surface through status; one session at a time.	packagePath (required for `st