uson1x/dsh-plugin-llm-verifier ↗★ 0
dsh-plugin-llm-verifier
LLM-as-a-Verifier for DeepSeek Harness (dsh): continuous reward signals via select / compare / track, using Monte Carlo repeated grading over decomposed criteria
AI 分析
核心用途是利用大模型作为评审器，对 Agent 的输出进行多维度、高精度的自动打分与对比。适合需要对模型生成质量进行自动化评估和对齐训练的开发者。
インストール
検証済み bundle がないか、互換性チェックに失敗しています。先にリポジトリの説明を読んでください。 README 全文を読む ↗
ドキュメント
README 全文を読む ↗Configuration
| Key | Default | Meaning |
|---|---|---|
provider | — (required) | Registered ctx.llm provider route to grade with |
model | — (required) | Model id on that route |
granularity | 20 | Integer score scale 1..G |
repetitions | 3 | Grading passes per criterion (K) |
temperature | 1 | Sampling temperature for the Monte Carlo estimate |
criteria | specification / output / errors | Array of { name, description } sub-criteria |
maxOutputTokens | 2048 | Output cap per grading call |
timeoutMs | 120000 | Deadline per grading call |
concurrency | 4 | Parallel grading calls |
tieMargin | 0.02 | compare margin below which the verdict is tie |
Cost note: verify_select makes N × C × K model calls (default 9 per candidate); verify_track makes steps × K.