agent-model-scout

Configuration

- id: scout
  name: "agent-model-scout"
  config:
    candidates: []            # explicit "provider/model" routes; empty = auto-discover
    lambda: 0.5               # cost weight: 0 ranks by quality only, 1 lets cost dominate
    topK: 4                   # max candidates probed when discovering
    maxProbeTokens: 500       # output cap for one probe
    probeTimeoutMs: 60000     # wall-clock budget for one probe
    probeTemperature: 0       # pin sampling so two runs are comparable
    reprobeGap: 0.15          # re-probe the top-2 when their gap is below this; 0 disables
    fixedOverheadTokens: 30000 # per-turn system prompt + tool schema cost of the real task
    priorMinDispatches: 2     # settled dispatches a route needs before its prior calibrates estimates

Candidate resolution order: --models flag → candidates config → auto-discovery (ctx.llm.listProviders() + listModels()) → the agent's current model.

Usage

Every successful probe ends in a picker card, and a picked model runs the task immediately — the loop is closed: probe → pick → execute.

/scout
/scout make the config loader support .toml and add tests
/scout refactor the session storage layer --apply
/scout ... --models deepseek-official/deepseek-flash,acme/big --top 2

Flag	Effect
(no task)	ask for the task, probe, pick, run
(task)	probe it, pick, run
--apply	switch onto the winner and run the task without asking
--dry	list the candidates that would be probed, spending nothing
--trust	rank from outcome-log priors alone, no probes; needs every candidate to have settled dispatches
--models a/b,c/d	probe exactly these routes
--top N	cap how many discovered candidates are probed
--lambda X	cost weight for this run (0 = quality only, 1 = cost dominates)

Picking the keep-current option runs the task without switching; dismissing the card runs nothing. The hand-off is agent.followup() with a frozen user message, so the task enters the session log as an ordinary turn and the recorded model selection takes effect on that turn's first step.

A chosen model that exposes reasoning efforts gets a second card: how hard it should think while executing (its own effort ids plus 默认, the model's default). The suggestion follows the plan — risky or many-stepped work earns the deepest effort, simple work the cheapest — and --apply applies the suggestion without asking. Efforts are discovered per model via ctx.llm.resolveModelInfo(); a model without an effort catalog skips the card silently, and the keep-current path leaves the effort untouched.

The picker is rendered through ctx.userQuestions.ask(), the same service behind the ask_user_question tool, so it appears as a normal question card. On a surface without a question UI (headless), the wizard refuses and the flag-driven forms still work.

--apply calls the native agent.model.select(), which is durable in the session projection and takes effect on the next step. The decision record is the command's own log entry plus the report text in command/done, so a replayed session still shows why the model changed.