uson1x/dsh-plugin-llm-verifier ↗★ 0
dsh-plugin-llm-verifier
LLM-as-a-Verifier for DeepSeek Harness (dsh): continuous reward signals via select / compare / track, using Monte Carlo repeated grading over decomposed criteria
AI Analysis
核心用途是利用大模型作为评审器，对 Agent 的输出进行多维度、高精度的自动打分与对比。适合需要对模型生成质量进行自动化评估和对齐训练的开发者。
Install
This plugin has no verified bundle, or compatibility checks failed. Read the repository notes first. Read the full README ↗
README
Read the full README ↗Configuration
| Key | Default | Meaning |
|---|---|---|
provider | — (required) | Registered ctx.llm provider route to grade with |
model | — (required) | Model id on that route |
granularity | 20 | Integer score scale 1..G |
repetitions | 3 | Grading passes per criterion (K) |
temperature | 1 | Sampling temperature for the Monte Carlo estimate |
criteria | specification / output / errors | Array of { name, description } sub-criteria |
maxOutputTokens | 2048 | Output cap per grading call |
timeoutMs | 120000 | Deadline per grading call |
concurrency | 4 | Parallel grading calls |
tieMargin | 0.02 | compare margin below which the verdict is tie |
Cost note: verify_select makes N × C × K model calls (default 9 per candidate); verify_track makes steps × K.