Guide Dog for DSH, powered by MiniMax

English | 简体中文

A dynamic Cordis plugin that gives DeepSeek Harness multimodal superpowers through the mmx CLI (MiniMax):

• Eyes for DeepSeek — MiniMax VLM (guide_dog_vision / guide_dog_inspect) describes images, so a model with no native vision input (e.g. DeepSeek) can still review frontend designs, figures, screenshots, and generated images.
• Hands for generation — images (image-01), video (MiniMax-H3 / Hailuo), speech (MiniMax TTS), music (music-3.0), text (MiniMax-M3), and web search.
• Web UI preview & playback — every generated file is served same-origin at /guide-dog/media/ and rendered inline in the conversation tool cards (, , ``), plus a Guide Dog settings page with auth status and a speak tester.
• Skill integration — guide_dog_speak reuses your existing audio-conversation and speech-mmx skill pipelines (text transform, CJK auto-detect, per-language voices, host playback), and falls back to raw mmx speech synthesize when the skill scripts are absent.
• Automatic invocation — a mounted system-prompt section (guide-dog-vision, order 110) tells the agent to auto-invoke the inspection tools for any job needing visual checks, especially when the active model cannot see images.
• Call mode (Phase 2, shipped) — hands-free, real-time voice conversation in the web UI: VAD / push-to-talk turn-taking, streaming sentence-level TTS with barge-in, consensus-first protection for write commands, progress announcements, and a unified floating dual-pill UI at the composer with zh/en i18n (details in "Phase 2 — call mode" below).
• Accessibility mode (Phase 3, planned) — an a11y config block is reserved (auto-narration, vision-cloud, summary-first); accessibility features are next on the roadmap and will be tested and rolled out after the call-mode shakeout.

Featured

Guide Dog is featured in dsh-recommend, a community-curated plugin directory for DSH. It passed the project's certification review and carries the gold certified badge at the top of this README; the score badge updates automatically on every registry sync.

Files

File	Purpose
plugin-host.js	Host half — source of record (tools, RPC, media route, prompt section, voice mode)
plugin-client.js	Client half — source of record (tool cards, settings page, voice cluster)
bundle/	Static web-profile bundle generated from the two halves (deploy/convert_bundle.py)
deploy/	convert_bundle.py (source → bundle) and publish.py (bundle → ~/.dsh/dsh-guide-dog + web profile registration)
package.json	Root manifest declaring the dsh.bundle (makes the repo installable via dsh plugin add)
README.md	This file
README.zh-CN.md	Simplified-Chinese version of this README

Install

Install straight from this repository with the standard DSH plugin command — the root package.json declares the dsh.bundle manifest, so the profile's pnpm layer resolves the package itself (no npm publish required):

dsh plugin --profile web add github:AtropinolTT/dsh-guide-dog

Deploy (static web-profile bundle — current)

1. Edit the source of record: plugin-host.js / plugin-client.js.
2. python3 deploy/convert_bundle.py — regenerate bundle/lib/.
3. python3 deploy/publish.py — copy to ~/.dsh/dsh-guide-dog, idempotently register in ~/.dsh/profiles/web (dependency link + bundles entry + node_modules symlink), remove the superseded autoload bundle.
4. Restart DSH (dsh web) — bundles are parsed at startup.

No dynamic plugin, no approval cards, no per-session instances: after a DSH restart the tools and voice UI come back with the profile itself. Full details and pitfalls in the "Restart recovery" section below.

plugin-source.js is a legacy dynamic-era artifact (both halves concatenated); kept for reference, not used by the current deploy flow.

Tools

Tool	Args	Returns
guide_dog_speak	text*, voice (auto), speed, language, playOnHost	{ok, url, voice, bytes} mp3
guide_dog_image	prompt*, aspectRatio, n, width, height, seed, promptOptimizer, watermark	{ok, urls[], files[]}
guide_dog_video	prompt*, model (MiniMax-H3 default), image, subjectImage, duration, ratio	{ok, url, taskId} mp4 (polls until done)
guide_dog_vision	image*, prompt	{ok, answer} VLM description
guide_dog_inspect	image*, focus (general/frontend/figure/screenshot/ocr), prompt	{ok, answer, focus} structured review
guide_dog_voices	language	{ok, voices[]}
guide_dog_music	prompt*, lyrics, instrumental, vocals, genre, mood, model	{ok, url} mp3
guide_dog_text	message*, system, model, maxTokens, temperature	{ok, text}
guide_dog_search	q*	{ok, results[]} (max 10)

* required

Auto-invoke contract (visual checks)

While the plugin runs, a system-prompt section instructs the agent:

• For visual checks (frontend design review, figure/plot/chart generation, screenshots, UI mockups, generated-image QA) it MUST call guide_dog_inspect (structured) or guide_dog_vision (general) on the produced image file before finalizing — never claim to have seen an image it has not inspected.
• Generated media is served to the user at /guide-dog/media/; the agent must include the returned url fields so the user can preview.
• Speech requests route to guide_dog_speak.

Example visual-check flow on DeepSeek:

1. (agent) create figure/screenshot file, e.g. chart.png
2. (agent) guide_dog_inspect { image: "chart.png", focus: "figure" }
          → structured review of axes/labels/readability/encoding
3. (agent) iterate the figure, re-inspect, then finalize with the url
4. (user)   previews chart.png in the web UI card

Media store & serving

• Media lives in ~/.dsh/guide-dog/.guide-dog/media — the global store under GLOBAL_ROOT = ~/.dsh/guide-dog (one instance for the whole web profile since 2026-08-16; no longer the per-workspace sandbox root — see "Restart recovery" below).
• Served by a same-origin prefix route /guide-dog/media with:
  • extension allowlist (jpg/jpeg/png/gif/webp/mp3/wav/m4a/ogg/mp4/webm),
  • basename-only lookup + traversal guard,
  • Accept-Ranges: bytes with real byte-range responses (video seeking),
  • 404/405/413/416 as appropriate.
• .index.json keeps metadata (prompt, voice, ts, kind) for the settings gallery (guide-dog/list-media RPC). A corrupt index is rebuilt from the directory.
• Files persist across plugin restarts; stopping/removing the plugin only removes the runtime registrations, never the files.

Skill integration (audio-conversation / speech-mmx)

guide_dog_speak honors the exact pipeline of your two skills:

1. ~/.agents/skills/audio-conversation/scripts/transform.py — markdown/code/URL stripping (falls back to a built-in JS transform when absent).
2. CJK auto-detect → per-language voice defaults (English_Trustworthy_Man / Chinese (Mandarin)_Gentle_Youth), same as the skill env contract. Explicit voice overrides; language boosts accents.
3. ~/.agents/skills/speech-mmx/scripts/mmx_tts.py speak --input … --out … (falls back to mmx speech synthesize).
4. Browser playback via the returned mp3 URL. With playOnHost: true the host speakers play it too — one file at a time (previous playback is terminated first), mirroring the skill's latest-only rule.

Env vars of the skills that still apply when set in the dsh process environment: AUDIO_CONVERSATION_VOICE(_EN/_ZH), AUDIO_CONVERSATION_SPEED, AUDIO_CONVERSATION_DIR, AUDIO_CONVERSATION_NO_PLAY, AUDIO_CONVERSATION_KEEP_FILES, TTS_GEN. Turn files keep the turn-NNN.mp3 naming convention.

Settings page

Settings → Guide Dog (id guide-dog):

• Auth — mmx auth status result with the key masked (sk-c…xxxx); never logged in full.
• Voice mode — global default on/off radios (per-session override lives on the small speaker button at the input's bottom-left).
• Voice input — STT engine select (whisper / sherpa / minimax), recognition language (auto/zh/en), input device select (defaults to the system default), and auto-send-after-recognition checkbox.
• STT — faster-whisper availability + version/python, and the whisper model select (base/small).
• Speak tester — text + voice selector (from guide-dog/voices), plays the mp3 in the browser.

Phase 1 — voice mode & voice input

Feature list

• Voice mode (host event-driven) — a host session/event listener watches assistant/message events, extracts the reply text (event.data.content blocks with type === 'text'), checks whether voice mode is effective for that session (session override else global default), and enqueues the TTS result ({url, key}) or error into a per-session voiceQueue. The client polls the queue every second and plays it with a module-level Audio object, or shows a bottom-right toast + beep for 6s.
• Voice cluster — conversation.input.left entry guide-dog-voice (order 30) at the input box's bottom-left, themed with DSH tokens (--dsw-alias-*), inheriting the app font:
  • small speaker icon — click toggles the per-session voice-mode override (guide-dog/set-config with voiceMode.sessions); hover tooltip shows "Voice mode: on/off · Global default: on/off".
  • language dropdown — recognition language detection (auto/zh/en).
  • mic icon — record → transcribe → insert (feather-style SVG; recording state pulses red with a second counter).
• Session-scoped playback — playback runs on a module-level Audio object, so switching sessions never replays or interrupts it: the current clip plays to the end unless a new playback task (a fresh queue entry from any session) overrides it.
• Mic voice input — the mic in the cluster: MediaRecorder with 1s timeslices, live second counter, maxSeconds auto-stop, language from the dropdown, and transcribe via guide-dog/transcribe. Recognized text is inserted into the input box with inputActions.setDraft(text) (auto-send via inputActions.submit() when configured). Error states: mic_denied, no_device, empty_speech, stt_failed, stt_timeout, engine_unavailable, insert_failed (never silent).
• Recorder page — sandboxed clients that cannot record in-page get a 🎙 Open recorder page link to the standalone /guide-dog/recorder page (GET serves a self-contained HTML recorder; POST /guide-dog/transcribe-upload accepts raw audio/webm, 20 MB cap, and runs the same transcribeImpl).
• Settings controls — the Phase 1 config blocks above, backed by guide-dog/get-config / guide-dog/set-config / guide-dog/status.

config.json schema

Lives at ~/.dsh/guide-dog/.guide-dog/config.json (auto-created from defaults; all keys optional, deep-merged over the defaults):

{
  "voiceMode": { "default": false, "sessions": { "": true } },
  "voiceInput": {
    "autoSend": false,
    "engine": "whisper",
    "language": "auto",
    "maxSeconds": 60,
    "whisper": { "python": "python3", "model": "small" }
  },
  "tts": {
    "voiceEn": "English_expressive_narrator",
    "voiceZh": "Chinese (Mandarin)_Gentle_Youth",
    "speed": 0.95,
    "format": "mp3"
  }
}

• voiceMode.sessions maps a session id to a boolean override; default is the fallback. The speaker button at the input's bottom-left toggles the current session's override.
• voiceInput.engine: whisper (only engine implemented; sherpa/minimax are reserved — selecting them returns engine_unavailable).
• voiceInput.maxSeconds forces the mic recording to stop.

STT engine (faster-whisper)

The whisper engine shells out to a bundled Python script (.guide-dog/scripts/whisper_transcribe.py) using faster-whisper:

pip install faster-whisper        # needs Python 3.8+; installs torch cpu wheels
python3 -c "import faster_whisper; print(faster_whisper.__version__)"

The host probes availability at startup and writes the result to .guide-dog/status.json (whisperAvailable, whisperVersion, whisperPython), shown in the Settings → STT row. Model choices: base (fast) / small (accurate); first run downloads the model weights.

Verification

node --check plugin-host.js && node --check plugin-client.js          # syntax
curl -s http://127.0.0.1:3080/guide-dog/recorder | head -5             # recorder page serves HTML
curl -s -X POST http://127.0.0.1:3080/guide-dog/api/guide-dog/status \
  -H 'content-type: application/json' -d '{}' | head -5               # status RPC (compat layer)
cat ~/.dsh/guide-dog/.guide-dog/status.json                            # whisper probe result

Manual checks (after deploy): click the speaker button (voice mode on, turns green) → send a message → the assistant reply is spoken automatically; switch sessions mid-playback → the clip continues to the end and is NOT replayed; use the mic button → recognized text appears in the input box; Settings → Guide Dog shows the Voice mode / Voice input / STT blocks.

Phase 2 — call mode

Feature list

• WebSocket-free dual channel — uplink is a whole-clip POST /guide-dog/call-transcribe (webm/opus, ≤20MB, reuses the Phase 1 transcribeImpl and the local whisper pipeline) → {ok, text, language, durationMs}; downlink is GET /guide-dog/tts-stream over a chunked HTTP stream (the host spawns mmx speech synthesize --stream --format pcm --sample-rate 24000 per sentence and pipes stdout incrementally into res.write; the client reads the stream with fetch().body.getReader() → PCM→WAV → seamless Web Audio scheduling). No new WebSocket protocol surface on the transport layer; browser and CLI reuse the same pipeline.
• Automatic VAD + push-to-talk (PTT) — default VAD (call.mode='vad'): MediaRecorder (audio/webm;codecs=opus, 250ms timeslice, continuous recording) + a parallel AnalyserNode energy detector (RMS ≥ vad.threshold; silence for vad.silenceMs ends an utterance; vad.minSpeechMs minimum speech; vad.maxSegmentSeconds per-segment cap) — speak-pause-speak automatically becomes two turns; the panel can switch to ptt push-to-talk (hold the mic to talk, release to send; VAD parameters do not participate in endpointing, only in interruption monitoring).
• Consensus-first (core interaction paradigm) — active only when call/a11y is on; typing mode keeps the Phase 1 behavior: a prompt soft constraint (guide_dog_call_consensus systemPrompt variable, conversational wording: understand intent first, ask when unclear, explain before writing/modifying and wait for the user's go-ahead) plus a mechanical hard guarantee (tools/pre-execute waterfall interception: write/edit and destructive-bash heuristics rm/mv/cp/truncate/dd/overwriting-redirect/git push etc. → unconfirmed returns `{kind:'deny', reason: 'n