dsh-pc-pilot

Usage

The plugin registers one global tool, computer. Typical flow:

1. computer { action: "list_apps" } — running apps with pids, exact app_identity / identity_key, window titles, hwnds and rects.
2. computer { action: "get_window_state", window: { id, app }, include_screenshot: true, include_text: true } — indexed accessibility tree with stable element_id, revision/delta metadata, a window screenshot and snapshot_id.
3. Act on the state — element actions include the snapshot_id from the same observation. Browser results return a reusable browser: { endpoint, tab_id } target that can be passed back unchanged; element actions can use either the raw browser_element token or the short @eN ref returned by browser_state / browser_observe.
4. Observe again only when state is stale/unknown, the target changed, or the next decision needs information you do not already have. Desktop element_index values remain bound to the get_window_state that produced them.

Action reference (57 actions)

Action	Purpose
list_apps / list_windows / list_displays	Enumerate apps with exact identity / per-app windows / display topology
get_app_identity	Resolve a process/window to stable Win32 path or packaged AUMID identity; lazily return product/version/company and optionally verified Authenticode signer evidence
get_window_state	Indexed UIA tree + stable element ids + revision/delta metadata + per-window PNG screenshot + document text
click	Coordinate click or snapshot-bound element_index click
set_value / type_text / perform_secondary_action / select_text	Element-level write, text entry, named UIA pattern, text-range selection
press_key / hold_key	Key chords and timed holds
scroll	Standard scroll_x/scroll_y (including target-window horizontal UIA scrolling), or legacy amount plus direction
move / mouse_move / mouse_down / mouse_up	Standard move plus raw mouse primitives
drag	Standard path of {x, y} points or legacy endpoints; element move (background) or real SendInput drag (foreground)
screenshot / zoom	Full-display or region capture; crop screenshot_path from a prior capture (path remains accepted by the runtime for compatibility)
switch_display / cursor_position	Default capture display; real cursor location
launch_app / wait	Launch an app behind the active work without activating it; the window stays normally renderable for WGC/UIA instead of remaining minimized. Registered Windows activation protocols such as ms-settings:display are supported. Delegated app launches return a target only when exactly one new window is safely identifiable; pause between actions
activate_window / minimize_window / close_window / get_window	Bring window to foreground / minimize it directly with Win32 (no title-bar coordinates) / request a graceful WM_CLOSE and verify disappearance (otherwise returns window_close_unconfirmed) / query fresh window geometry & metadata
read_clipboard / write_clipboard	Clipboard round-trip
browser_tabs / browser_state / browser_observe / browser_history / browser_back / browser_forward / browser_wait	Manage exact tabs; browser_observe gives compact ref-only semantic state while browser_state keeps raw tokens for compatibility; navigate history and wait on page readiness/URL/text without synthetic sleeps
browser_events / browser_downloads	Cursor-based console/network/lifecycle evidence; Chromium download progress and completed AI-profile files
browser_shutdown	Close the entire browser only when this PC-Pilot tool instance launched it
browser_click / browser_type / browser_replace / browser_key	Operate a raw token or compact @eN ref from the latest state/observe result; stale or changed targets are rejected
browser_click_point	Click browser-viewport coordinates only when bound to an exact screenshot_id; both explicit observations and post-action screenshots remain chainable for 30 seconds while the tab/document/URL identity is unchanged
browser_upload	Select 1–20 explicit absolute local files on an already-observed `` and verify the browser received them; does not submit the surrounding form

Key parameters

Parameter	Default	Notes
dispatch	background	UIA patterns + window messages; never steals focus. foreground uses real SendInput — pick it per task only when the user asked for real control or the essential action has no background path.
activate	false	launch_app only: normal foreground launch. The default keeps the new window renderable but non-activated and places it behind the active work; use true only when the user explicitly wants it brought forward.
overlay	true	Show the click-through cursor at each action point; it auto-hides 3 s after the last action.
include_screenshot	true	Capture a per-window PNG in get_window_state.
include_text	false	Include the indexed accessibility tree and document text when an element action is needed. On a window-targeted wait, include it in the post-wait observation to check application readiness without a second round-trip.
wait_for	—	On a window-targeted wait, wait for accessibility_present (any UIA descendant) or accessibility_available (a complete UIA tree). A timeout is an explicit, retry-safe wait_condition_timeout.
app	—	pid number, process name, or window-title substring; same-titled duplicate windows are rejected unless window_index or hwnd identifies one, while differently titled windows of one app auto-resolve and return chosen_hwnd.
identity_key	—	Exact application identity returned by discovery. Packaged apps use aumid:; Win32 apps use win32:. Prefer it for continuation targeting once an app is known.
verify_signature	true	get_app_identity only: verify Authenticode signer/publisher and cache the result by executable path.
snapshot_id	—	Required for desktop element actions; use the id from the latest get_window_state { include_text: true }.
element_id	—	Stable UIA identity returned by get_window_state; optional for normal actions, required for safe element remapping during foreground_once recovery.
expect	—	Optional postcondition: verify window state, accessibility change, element value, text, browser URL/text/readiness, or completed download after the action.
when	—	Optional precondition using the same condition vocabulary as expect; checked before dispatch, with default timeout_ms: 0. If unmet the action is not_executed. Useful as a lightweight gate inside short actions batches.
recovery	none	foreground_once retries only a conclusively non-executed background_unavailable action. Unknown outcomes are never replayed.
browser / browser_endpoint / tab_id / browser_element / event_cursor	—	Prefer the reusable browser: { endpoint, tab_id } target returned by PC-Pilot; the separate endpoint/tab fields remain compatible. browser_element accepts a raw semantic token or compact @eN ref.
x / y	—	Window-local pixels with app/hwnd, matching the Computer Use coordinate model; screen coordinates without a target. Set coordinate_space: "screen" only for an explicit absolute click.
button / click_count / keys	left / 1 / —	Mouse button and legacy click repetitions; keys supplies standard keypress chords and mouse modifiers. Foreground mouse actions and validated native-window background clicks preserve the modifier state; unsupported background paths report background_unavailable.
actions	—	Ordered batch of action objects (maximum 20). Steps may use when before dispatch and expect after dispatch; the batch stops at the first unmet/failed/uncertain step. Intended for short predictable sequences, not scripted branching.
display	primary	1-based display index for screenshot / switch_display.