@lgquan/dsh-voice-assistant

English | 中文

Consumer that binds one attached voice transport to the current durable Session. In speech-shell mode, a completed transcription becomes followup when no voice task is active and steer while one is running in that Session. In frontend-agent mode, a route_transcription call gives the selected frontend model the recent durable Voice conversation and current transcription. It answers ordinary conversation directly, or converts tool-requiring work into a validated background delegation. While a task is active, later transcriptions update that exact task.

realtime_delegation allocates the authoritative VoiceTaskId, ensures a background task Session, and returns the id on the wire as delegation_id. With the bundled continuous policy, one source Voice Session creates or resumes one fixed ordinary task Agent Session; the compatibility isolated policy creates a Session per delegation. The task Agent inherits the Voice Session's workspace, working directory, preset composition, provider, and model. Its task message is a envelope containing the id, self-contained input, and optional transcript delta; the Voice Session records `voice/task-delegated` with the target `SessionId` for navigation. `send_task_message` sends an exact-id through that task Agent's steer, and cancel_task calls that Agent's cancel({kind: 'user'}). Unknown, terminal, and cancelling ids return typed rejections without Agent mutation; dsh-voice suppresses duplicate provider call ids before this consumer runs.

The plugin installs send_voice_message and its guidance only in the active task Agent's scope. The bridge created that exact target, so the backend tool exposes no project listing or selection. The Agent emits structured progress | result | warning | error | question events with complete factual detail; the tool has no Agent-authored speech field. Stage events may repeat when meaningful. Final result is accepted once and held until the authoritative turn succeeds. For every reported event, the Voice layer gives an independent model the original user request (including accepted updates), the event type, and its complete detail. Stage rewrites stay concise, while final length adapts to the request rather than a fixed token cap. Each rewritten event is submitted as one coherent UI message and one TTS response; Edge TTS retains internal sentence splitting without exposing separate bubbles. A question leaves the task in waiting-user; the reply opens the next turn in the same task Agent Session. Failure or cancellation discards a buffered result. If a successful turn omits a result, the bridge rewrites the last assistant text; when rewrite support is unavailable it uses a safe direct fallback, and only a turn with no usable output uses completedAnnouncement.

The exact identified task message is associated with its turn through agent/inbox/claimed; the corresponding durable turn/end is the task terminal. Every TaskObservation is appended to the source Session as voice/task-observation before provider delivery. Provider ASR and output-text lifecycles become durable utterance start/end events; deltas remain live browser state. The provider external-text projection makes spoken text part of that same durable assistant history and marks it completed only after browser playback finishes. These plugin-owned records are required-on-read: the plugin registers their types with KNOWN_SESSION_EVENT_TYPES at load, because DSH core has no generic skip-unknown-plugin-event registration surface yet. Speech-shell mode requests external-text speech for assistant output; frontend-agent mode independently rewrites reported stage events and terminal results. Terminal mapping is completed → completed, aborted → cancelled, and every other end reason → failed. Voice disconnect detaches transport only, keeps the task Agent and scoped reporting tool while its task is active, continues completing validated provider commands, and queues bounded observations for a same-process reconnect. Service shutdown persists an active task as interrupted; recovery announces the last spoken progress but never replays the task automatically.

Model Experience

Voice task observations

What the model sees

Speech-shell transcriptions enter the Voice Session Agent as human messages. Frontend-agent work enters the fixed background task Agent as a human delegation envelope because the provider model selected its wording. Only that task Agent receives send_voice_message; the voice provider receives no dsh business-tool schema.

Token effect

Every new frontend utterance uses one lightweight routing call. Direct conversation completes in that call and creates no background Agent task. Delegated work additionally consumes task-model tokens for execution and one auxiliary rewrite call per reported event. Local ASR/TTS add no model tokens.

KV Cache effect

Each admitted delegation or update extends normal task-Agent history. The complete voice/agent-binding-state record lets the continuous policy resume that history after restart; provider conversation state does not alter the task Agent's reusable history.

Known Limitations and Deferred Work

• The first version supports one active task per Voice Session.
• Observation buffering survives a transport reconnect only inside the current process. Durable utterances, the complete Agent binding, and interrupted-task position survive restart, but provider model process state does not.
• Voice events are required-on-read. A DSH build without this plugin refuses to load a Session containing voice/* events; installing the plugin registers the types at load. A generic upstream registration surface would remove this coupling.