dsh-eval-regression

A small, deterministic regression-evaluation plugin for DeepSeek Harness.

It registers evaluate_golden_output, a model-callable tool that compares supplied candidate output against required and forbidden fragments. It does not call a model, persist data, or claim semantic correctness. Its job is repeatable pass/fail evidence, not vibes-based architecture in a trench coat.

Why

Agent changes routinely regress answers that appear superficially acceptable. A stable corpus of expected fragments gives a cheap, transparent signal for release smoke tests and replayed transcripts:

• required fragments catch omissions
• forbidden fragments catch known bad claims or unsafe fallbacks
• per-case reports make failures reviewable
• deterministic scoring is suitable for CI thresholds

Install as a DSH plugin

dsh plugin --profile 
 add github:aryswisnu/dsh-eval-regression

The package is a DSH bundle. Its cordis.patch.yml registers the tool automatically after the profile's base tool runtime.

For local development:

git clone https://github.com/aryswisnu/dsh-eval-regression.git
cd dsh-eval-regression
npm install
npm run build
dsh plugin --profile 
 add .

Run a version-controlled suite in CI

The plugin also ships a small CLI. It reads a JSON suite, prints an evaluation report to stdout, exits 0 when every case passes, exits 1 when any case fails, and exits 2 for invalid input or usage errors.

{
  "suite": "release-smoke",
  "cases": [
    {
      "id": "grounded-answer",
      "actual": "The result is 42. Source: benchmark.csv",
      "includes": ["42", "Source:"],
      "excludes": ["I cannot verify"]
    }
  ]
}

npx dsh-eval-regression suites/release-smoke.json
# or, from this repository:
npm run evaluate -- suites/release-smoke.json

The report includes total passed and failed cases, a 0..1 score, and case-level missing or forbidden fragments. This makes the evaluation corpus ordinary, reviewable source code and makes a failed expectation fail the CI job.

Tool example

{
  "suite": "release-smoke",
  "cases": [
    {
      "id": "grounded-answer",
      "actual": "The result is 42. Source: benchmark.csv",
      "includes": ["42", "Source:"],
      "excludes": ["I cannot verify"]
    }
  ]
}

The canonical result includes total passed and failed cases, a score, and each case's missing or forbidden fragments.