dsh-bench

English · 中文

Run one task across several model/config variants in DeepSeek Harness and compare the trajectories side by side.

The plugin registers one tool, bench_run. Each trial is a fresh one-shot spawn subagent that sees only the task text — no parent history — so the arms are comparable.

Why

The plugin ecosystem has plenty of tools that observe a single session — usage dashboards, context breakdowns, trajectory diagnostics — and none that compare two. Without a comparison there is no way to answer whether a cheaper model, a smaller tool surface, or a shorter output budget actually changes the outcome; there are only opinions about it.

Every number here already exists in the harness. The plugin drives the matrix and folds the results:

what	where it comes from
steps, model wall time	sessionStats session-projection unit
tokens	ctx.tokenMeter.measure(session)
tool calls and mix	tool/call events in the child's durable log
stop reason	SubagentResult.stopReason

Mount

Nothing in $DSH_HOME needs editing — the plugin ships its own overlay. Copy the template and point it at wherever you cloned this repo:

cp bench.patch.yml.example bench.patch.yml
# then edit the `name:` line to your own absolute path
dsh --profile web --patch /path/to/ds-bench/bench.patch.yml

Verify the composition first — this boots nothing and costs nothing:

dsh --profile web --patch /path/to/ds-bench/bench.patch.yml --dump-config

Path gotcha. A relative name: inside a --patch overlay resolves against the profile directory ($DSH_HOME/profiles/), not against the patch file — unlike an agent preset, where it resolves against the preset directory. A plugin living outside the profile must therefore use an absolute specifier, and on Windows Node's ESM loader accepts an absolute path only as a file:// URL.

The plugin is loaded from outside the profile, so it sits outside the HMR watch root: changes need a restart, not a reload.

Use

bench_run defaults to a dry run and returns the plan without sending anything. Pass confirm: true to spend tokens.

{
  "task": "Read package.json and list every script.",
  "variants": [
    { "label": "pro" },                                       // inherits the caller's route
    { "label": "flash", "model": "deepseek-v4-flash" },
    { "label": "flash-capped", "model": "deepseek-v4-flash", "maxTokens": 1024 }
  ],
  "repeat": 3,
  "confirm": true
}

Ran 6/6 trials.

| variant | provider/model | runs | completed | steps | tool calls | tokens | llm ms | tool mix |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| pro   | deepseek/deepseek-v4-pro   | 2 | 2/2 | 7 | 8 | 24448 | 8124 | glob×6, read×4, grep×2 |
| flash | deepseek/deepseek-v4-flash | 2 | 2/2 | 4 | 4 | 11047 | 3011 | read×3, glob×2 |

There is no UI: bench_run is a model-facing tool, so you invoke it by asking the agent for it in the composer.

Aggregates are medians and a completion count, never means. Trajectory outcomes are closer to Bernoulli than Gaussian, and a mean over a bimodal sample reports a value no run produced.

Config

key	default	meaning
providerName	spawn	Subagent provider. fork would inherit the caller's conversation and destroy comparability.
maxTrials	24	Hard cap on variants × repeat per call.

Deliberate limits (P0)

• Sequential trials. Concurrency would pollute llmMs with queueing delay and make the wall-time columns incomparable.
• No preset axis. SubagentStartRequest carries no preset override — children join their parent's preset — so that axis needs ctx.agents.create() instead of the subagent seam.
• No context-ablation axis (suppressing agent/pre-step injections per arm) and no pass/fail check command yet. Both are next; the ablation hook is proven to work by $DSH_HOME/.agent-presets/anchored-standard/tool-bootstrap.mjs.
• First-trial failures are fatal. A bad provider name, model route, or depth budget fails the whole call rather than burning the matrix against the same fault. Later failures are recorded as notes and the run continues.

Status

P0, verified against a live 4-trial run (2 arms × repeat 2) on 2026-08-15.

What that run established:

• Composition, module load, tool registration, JSON-Schema acceptance at prompt assembly, output validation, and report rendering all work.
• No approval deadlock. The children executed 12 tool calls (glob, read, grep) with no gate stalling the parent — the delegated policy carries down as child-agent.ts implies.
• One real bug, since fixed: Cordis gates service access, not merely plugin activation. Reading ctx.tokenMeter without declaring the dependency throws cannot get property "tokenMeter" without inject, so a defensive try/catch degraded every metric column to 0. Optional seams now arrive through ctx.inject(…); a composition lacking them still reports tool mix and completion counts, with a note naming what is missing.
• Retry backoff dominates wall time when a route is broken. A failing arm cost roughly 5 of the run's 6 minutes: 2 repeats × 2 llm/retry attempts each, with backoff between them. A per-trial timeout belongs in P1.

That run also produced the tool's first genuine finding, which is the whole point: one arm scored 0/2 completed with zero tool calls, its children ending in turn/end → error, "Stream ended without finish_reason" (TRANSPORT). A broken provider route is invisible in normal chat — it merely feels slow — but next to a working arm it is unmissable.

Roadmap

• P1 — per-trial timeout, a deterministic check command for pass/fail, and a separate failure-reason column (a transport failure and an unfinished task currently both just lower the completion count).
• P2 — the context-ablation axis: suppress agent/pre-step injections per arm to price what each injected context section is actually worth.
• P3 — trajectory diff, a preset axis via ctx.agents.create(), and concurrency.

License

MIT