dsh-plugin-watchdog

Host-side crash/restart watchdog for dsh. When the dsh server process dies — a crash, OOM, kill -9, or a restart — every running agent is interrupted and the work simply stops. The persisted session logs survive, but nothing picks the work back up until a human reopens the sessions. This plugin closes that gap:

• No browser needed. It is a host-only Cordis plugin; it runs inside the dsh server and works whether the GUI is open, closed, or never opened.
• On restart it checks recently active sessions and resumes the ones whose work was genuinely interrupted — not the ones that were deliberately stopped.
• It picks up where they left off: it resumes the persisted session via ctx.agents.resume, then either re-arms the active goal (the goal-round driver queues the next `` itself) or wakes the agent with a "continue where you left off" follow-up prompt. Crash-recovery closers (TOOL_OUTCOME_UNKNOWN / TOOL_NOT_STARTED) are already in the resumed history, so the model verifies uncertain side effects before retrying.

Install

The plugin is a host-only Cordis plugin resolved from your dsh profile's package store. The profile's dsh plugin command forwards to pnpm, so any pnpm-supported git spec works.

From GitHub

dsh plugin --profile web add github:davidgereb/dsh-plugin-watchdog

This also installs the host half's dependency, dsh-lib-context-injection (declared as a git dependency in this package's package.json).

From a local checkout

# 1. build (regenerates lib/index.js + lib/client.js from src/)
node scripts/build.js
# 2. make the local package resolvable by the profile
dsh plugin --profile web link /path/to/dsh-plugin-watchdog
# 3. same for the lib dependency
dsh plugin --profile web link /path/to/dsh-lib-context-injection

Then add the loader row to the profile patch file ($DSH_HOME/profiles/web/cordis.patch.yml):

# ── Watchdog: resume sessions interrupted by a server crash/restart ──
# Host-only plugin (no browser half). On boot it resumes recently-active
# sessions whose agents were interrupted by an unclean server stop and
# continues their work (goal re-arm or a continue follow-up).
- insert:
    - id: watchdog
      name: dsh-plugin-watchdog

The profile patch is read at server start, so the row takes effect on the next dsh web start — which is also the first real test of the watchdog.

How it decides what to resume

1. It tracks sessions while the server runs

A small persisted registry ($DSH_HOME/storages/watchdog.json) records every session the watchdog sees and when it last did anything:

• every durable session/event touches the session's lastActiveAt;
• agent/status → running (and a periodic poll of ctx.agents.list()) also records lastRunningAt and clears cleanStop.

2. A graceful stop is a deliberate stop

On a graceful shutdown (Ctrl+C / SIGTERM — the plugin's effect disposer and the loader's exit event both run), every tracked session is marked cleanStop. A crash or kill -9 runs neither, so cleanStop stays false.

3. The boot scan resumes only real pending work

After the loader tree settles, the watchdog looks at every tracked session that is recently active (lastActivity within staleMs, default 24 h), not cleanly stopped (unless resumeAfterGracefulShutdown), and not already live. It also does a one-time disk scan of the sessions directory for recently-touched logs it has never seen (so it also helps sessions that were running before the plugin was installed).

For each candidate it reads the session log non-mutatingly (sessionPersistence.inspect) and checks for durable pending work:

Signal	Means	Present when…
interrupted-turn	a turn/end { interrupted } closer is in the log — the persistence layer only emits these when it closed a turn the crashed process left open	crash mid-turn
pending-inbox	the durable agent/inbox/spliced projection is non-empty	prompts queued but never claimed
active-goal	the latest goal/change is phase active with roundsStarted / The resume mechanics (route derivation,installModelSelection` wrap, and the

follow-up message injection) live in the shared dsh-lib-context-injection library, which dsh-plugin-cost-lens also uses for its scheduled/off-peak sends.

Browser half (lib/client.js): the Settings → Plugins card reading/writing /watchdog/api.

Caveats

• Resume is a fresh continuation, not mid-turn time travel. A crash-mid-turn session is closed by the persistence layer's crash repair (synthetic tool/result + turn/end { interrupted }), and the model continues from there — it sees the repair markers and our follow-up prompt, and verifies uncertain side effects before retrying. There is deliberately no partial-turn rewind (that is a persistence-layer limitation, not this plugin's).
• Single-writer sessions. Only one dsh process may write a session log at a time. Do not run two servers against the same $DSH_HOME and expect both to resume the same sessions; the losing resume is caught and logged.
• Deliberate restarts are treated as deliberate stops by default; flip resumeAfterGracefulShutdown if you want graceful restarts to continue too.
• The watchdog only tracks what it has seen. The untracked disk scan covers sessions that ran before install, but it is bounded (scanUntrackedMax) and mtime-based — very large old logs are not scanned.

Testing

node scripts/build.js
node scripts/smoke-test.mjs      # pure event-analysis logic (30 checks)
node scripts/mock-boot-test.mjs  # full apply() orchestration against a mock ctx (20 checks)

Compatibility. Tested against dsh 0.1.0-rc.6 on Node.js v24.19.0 (dsh web profile). Older or newer dsh releases may change the internals this plugin hooks into — check the changelog before upgrading.

---

⚠️ AI-generated, provided as-is. This project was written with the assistance of an AI. It is provided AS IS without warranty of any kind, express or implied. The author cannot be held responsible for any damage, data loss, or misbehaviour that results from using it. Use at your own risk.