Cortex — Multi-Model Orchestration & Evaluation for DeepSeek-Harness

English | 中文

Cortex is a DeepSeek-Harness plugin that turns the host's primary (strong) model into a high-leverage supervisor. The supervisor plans, decomposes, verifies and accepts; routed executor models (small/cheap models such as Qwen3.8-9B) and tools do the heavy lifting. With a recursive task tree, dynamic routing, four-stage quality gates, budget control, failure recovery, policy reuse and per-dimension model evaluation, Cortex keeps quality stable while driving the unit cost of successful tasks down.

The supervisor is the host primary model that drives the cortex_* tools (e.g. deepseek-v4-pro in the bundled registry), and executors are the models the dynamic router picks per subtask (L1/L2 tiers, e.g. Qwen3.8-9B).

Suggested repository topics (GitHub): dsh-plugin · dsh · cordis · llm-orchestration · multi-model · model-routing · quality-gate

---

Highlights

Capability	Entry	Notes
Task intelligence	cortex_start	Profile → fingerprint → strategy match (≥0.90 reuse / 0.70~0.90 light-check / # the repo root
npx @deepseek-ai/dsh plugin --profile web add . # anchors to this dir (or pass the absolute path)
npx @deepseek-ai/dsh web

**Remove:** `npx @deepseek-ai/dsh plugin --profile web remove dsh-cortex`.

The shipped bundle patch inserts the `cortex` row with `uiEnabled: true, uiPort: 8788` (console listens on loopback only). `stateDir` and `modelsFile` use built-in defaults (`$DSH_HOME/storages/cortex` and the bundled `config/models.yaml`) — anything can be overridden by an `- id: cortex` config row in your profile's own `cordis.patch.yml` (it applies after bundle layers).

## Typical Flow (agent view)

cortex_start → (cortex_strategy_match) → cortex_decompose T0 → cortex_execute [node_ids | recursive] → cortex_review (batch accept) → cortex_recover (on failures) → cortex_report (KPI) → cortex_policy (settle)

## Operations & Commands

### 1. Agent tools (12 — the supervisor calls them autonomously)

| Tool | Purpose | Key parameters |
|---|---|---|
| `cortex_start` | Create a task (profile → fingerprint → strategy match → budget pool) | `goal`, `profile{task_type, domain, complexity, risk, quality_requirement, budget_limit, input_tokens?, expected_output_tokens?, latency_requirement_ms?, language?, capability?}`, `input` |
| `cortex_strategy_match` | Policy-center lookup (3-tier reuse verdict) | `profile` |
| `cortex_decompose` | Split a parent node (safety limits / DAG / replace) | `task_id`, `parent_id?`, `replace?`, `children[{goal, task_type, capability, output_schema, quality_target?, tool_allow?, depends_on?, type?}]` |
| `cortex_execute` | Route & execute (Gate0/1[/2], cache, accounting) | `task_id`, `node_id?`/`node_ids?`, `recursive?`, `mode?` (costFirst|balanced|qualityFirst), `multi_vote?`, `max_parallel?`, `force_models?`, `evaluate?` (none|gate2) |
| `cortex_review` | Gate3 batch acceptance (decisions carry quality/self-check snapshots) | `task_id`, `decisions[{node_id, status, reason_code?, missing_items?, next_action, recommended_depth?, confidence?}]` |
| `cortex_recover` | Recovery decision tree (retry/switch/upgrade re-run in-engine) | `task_id`, `node_id?` |
| `cortex_policy` | Settle a policy from a successful task | `task_id`, `note?` |
| `cortex_report` | KPI report (per task or global) | `task_id?` |
| `cortex_models` | Model registry / profiles / routing scores / drift | `task_type?`, `capability?`, `reset?` |
| `cortex_evaluate` | Model evaluation (7-dim × capability profiles, value/Pareto) | `cases?`, `use_saved?`, `models?`, `task_type?`, `quality_target?` |
| `cortex_eval_cases` | Eval-case library (page + conversation shared) | `action?` (list|add|remove), `input`, `goal`, `capability`, `expected`, `output_schema`, `id` |
| `cortex_capabilities` | Capability catalog (append-only) | `action?` (list|add), `name`, `description?` |

### 2. Slash commands (`/cortex `)

| Command | What it does |
|---|---|
| `/cortex status` | Engine state: tasks (running), nodes, attempts, policies, models, **discipline status** |
| `/cortex task ` | Task tree (profile/strategy/budget/nodes) |
| `/cortex trace ` | Execution trail (attempts + decisions) |
| `/cortex kpi [id]` | KPI report (global or per task) |
| `/cortex models [reset]` | Registry listing; `reset` clears evaluation feedback |
| `/cortex policies [disable|enable ]` | Policy library + controlled enable/disable (rollback = re-enable) |
| `/cortex model-pref [set  [reason]\|clear\|show]` | Set/clear the **preferred executor model** with **preference content** (`reason` = which tasks it suits): routing adopts it when the task scene is *covered* by the content (>=50%) + active + quality  · /api/events (SSE live)
POST /api/models/            {enabled?: boolean, preferred?: boolean, reason?: string}   // reason = preference content (required on the page)
     /api/matrix/reset           (clear evaluation feedback)
     /api/eval-cases             {action: add|remove|save, ...}
     /api/capabilities           {name, description?}
     /api/evaluate               {models?, cases?, use_saved?} (real background run)

4. Runtime lifecycle

• Mount: apply → registry load (static YAML + persisted overrides + model states) → heartbeat stateDir/mounted.json → optional console → usage-settle timer (60s, zombie sweep every 5 ticks) → dynamic model discovery.
• Per turn: orchestration-discipline context is injected per conversation session (suppressed only when that session has a flow active within 15 min); other sessions' leftover flows never suppress.
• Unmount / restart: effect dispose stops the console (active SSE connections destroyed — no hang) and clears timers; everything durable lives in the state dir (tasks/nodes/policies/profiles/model states/rate) and rebuilds on restart (last-write-wins).
• Self-healing: zombie tasks (running, 4h silent) are finalized automatically; evaluation feedback and drift baselines survive restarts; /cortex reset wipes state files for a fresh start.

Model Profiles (how they update)

• Evaluation write-back (cortex_evaluate / one-click UI run): per task-type quality EMA + success rate + cost/latency + 7-dim capability profiles (dims_profile), persisted in profile_overrides.json (drift baselines anchored at first measurement).
• Execution feedback (round 69): every real node execution (non-cache) updates deterministic counters — attempts/complete/failed + failure-type & task-type distributions, persisted in the same file; when samples >= 3 (EXEC_PROFILE_MIN_SAMPLES), routing uses the measured success rate instead of the static one (the more it is used, the truer the profile, the better the routing). Quality scores still come from evaluations only (Gate1 self-checks never pollute profiles).
• POST /api/matrix/reset or /cortex models reset restores static baselines (evaluation + execution feedback).
• Measured-only scores (round 72): the capability matrix shows a value only for verified capabilities (evaluation/execution write-back keys, tracked in measured); static declarations and defaultQuality never masquerade as verified scores — unverified cells display "no measured data".

Sub-Model Capabilities & Media Support

Executors (routed sub-models) are self-contained worker agents. What they can and cannot do:

Capability dimensions include media tags: image_analysis (declared by vision-capable models — image tasks route & filter precisely by it) and audio_video_analysis (catalog dimension; no model declares it yet — matrix stays empty until a media-capable model/MCP is configured). Decompose image subtasks with capability: ['image_analysis'] (not document_analysis) for exact routing.

Capability dimensions include media tags: image_analysis (declared by vision-capable models — image tasks route & filter precisely by it) and audio_video_analysis (catalog dimension; no model declares it yet — matrix stays empty until a media-capable model/MCP is configured). Decompose image subtasks with capability: ['image_analysis'] (not document_analysis) for exact routing.

Dimension	Supported
Input	text, files (read/glob/grep), images (read_image / MCP browser), URLs (web_search/web_fetch, read-only) — also base64 images (data:image/...;base64 or {image_base64, mime}) auto-decoded to a real file for the executor
Tools (agent mode)	read glob grep read_image + web_search/web_fetch (read-only, default; override via executorExtraTools) + mcp__* (any configured MCP server — e.g. Playwright screenshots) + skill + business-plugin prefixes (default tssdp_)
Policy	mediaPolicy: auto (default — media input forces agent mode + contract), reject (explicit fast-fail with diagnosis), pass_through (hand to MCP)
Output	structured JSON + optional top-level artifacts [{path, mime}] — engine validates existence & magic bytes (png/jpeg/webp/gif/mp4/mp3/wav/pdf); invalid → node failed with audit trail
By default denied	write/edit/shell (pwsh/bash/run_code) and re-orchestration (subagent/workflow/cortex_*/…) — an explicit per-node tool_allow swaps in a custom allow list for artifact-producing tasks
Honest limits	DSH provider modalities are text + image only — audio/video understanding requires a media-capable MCP/multimodal route; otherwise the executor returns the {"unsupported": true, "reason": ...} contract (never invents content), or mediaPolicy: reject fails fast with a clear diagnosis

Example flows: image analysis → cortex_execute on a node with image references (agent mode + read_image/MCP); screenshot/poster generation → Playwright MCP + artifacts declaration (magic-byte validated); audio/video processing → only with a media MCP configured (else explicit diagnosis, classified INPUT_FAILURE).

Preferred-model fallback (round 70/73): when a preferred executor (e.g. a text-only L1 model) is handed an image task, it runs first (attempt is recorded). The executor now pre-checks modality deterministically (modelAcceptsImage via host llm.resolveModelInfo): a model that does not declare image input fails fast with model lacks image modality (no generation — a text model must never be allowed to fabricate read-image evidence; observed in production: invented 800x600 / white-pixel result for a 1x1 red PNG). The recovery engine detects the gap and switches directly to a vision-capable candidate — no unnecessary decompose/upgrade round-trip.

Host prerequisite for vision: the vision model must declare inputModalities: [text, image] in the host's llm-deepseek.models entry (e.g. ~/.dsh/settings.yaml) — otherwise read_image refuses with "model does not declare image input" (the catalog default for undeclared entries is [text]).

Host prerequisite for vision: the vision model must declare inputModalities: [text, image] in the host's llm-deepseek.models entry (e.g. ~/.dsh/settings.yaml) — otherwise read_image refuses with "model does not declare image input" (the catalog default for undeclared entries is [text]).

Cost Model

• Executor models (sub-models): exact TokenUsage from subagent/chat sessions (billed = input + cache read/write), priced from the registry — true token accounting.
• Supervisor (primary model): real session-model usage via DSH tokenMeter/sessionProjections (fallback estimate), plus a "vs direct primary-model run" baseline saving in cortex_report.
• UI-triggered evaluations run without a supervisor (chat path) and are not counted as supervisor calls.

Development

npm test                 # node --test (engine/service/ui unit + smoke) — 141 tests
node tools/verify-post-rest