jmche/dsh-llm-verifier-pro ↗★ 0
dsh-llm-verifier-pro
LLM-as-a-Verifier for DeepSeek Harness (unified): fine-grained reward tools (verify_compare / verify_select / verify_track) with Probabilistic Pivot Tournament and vLLM/SGLang prefill, PLUS the Best-of-N conversation mode with a Web settings panel. Merged from dsh-llm-as-a-verifier (TaurenMountain) and @aispin/plugin-verifier (Aispin).
AI Analysis
核心用途是通过多候选采样和 LLM 评估提升回答质量。适合对生成结果有高精度要求、希望利用 Best-of-N 算法让模型进行自我验证和筛选的用户。
Install
npx -p @deepseek-ai/dsh dsh plugin --profile web add github:jmche/dsh-llm-verifier-proREADME
Read the full README ↗Configuration
Zero-config default: with no explicit baseUrl / apiKey / model (and
the panel's Verifier fields empty), the verifier follows the session —
same provider route, endpoint and model as the conversation. Turning on
Best-of-N alone gives the paper's self-verification experience: candidates
are sampled as variants of the conversation's own model, and that same model
grades them. The endpoint for the session provider is read from its settings
namespace (llm-pi-ai.providers. style). Only when the session
provider is unknown does the resolution fall back to:
plugin config → the verifier settings section →
session provider endpoint → OPENAI_BASE_URL / OPENAI_API_KEY /
DEEPSEEK_API_KEY → api.deepseek.com.
The verifier must sit on an endpoint that returns token-level logprobs
(vLLM, SGLang, OpenAI, DeepSeek, and modern Ollama all do; a plain gateway
that strips logprobs will not). Non-DeepSeek servers get the optional
vLLM/SGLang prefill pass so score tags land exactly at the label position.