# model_proctor
A deterministic control plane for coding agents on [Kimi Code CLI](https://github.com/MoonshotAI/kimi-cli):
task-owning workers dispatched through a thin runner, leader-executed verification with tree-bound
acceptance receipts, wire-metered cost accounting, and a pre-registered evaluation harness.
**The name.** A *proctor* assigns the exam, watches the clock, and grades it objectively — the model
never marks its own work. That is the architectural invariant this repo converged on: deterministic
evidence outranks every model, and acceptance is bound to the exact tree that was verified.
(Historical names: `robot_lockstep_ballast` — *lockstep* for fixed rungs, *ballast* for the cheap
fleet — then briefly `kimi_router`; renamed 2026-08-25. *Static cascade* was the frozen research
**pattern** that preceded the runner; its artifacts were removed 2026-09-25 per owner directive —
the pattern's history remains documented in the research-lineage sections below.)
---
## Status update (2026-09-09, corrected 2026-09-10): the gated path stays — flash-first with evidence-driven escalation
The full record is [`policy/FINDINGS-2026-09-09.md`](policy/FINDINGS-2026-09-09.md); the
conversion is tracked in [#96](../../issues/96). The short version:
| Value pillar | Verdict (measured) |
|---|---|
| Routing by task shape | **Zero on the measured corpus** (STOP fired twice, #30/#91 — bounded ~30s tasks, quality-tied tiers); **long-task economics favor cheap-first + escalation** beyond the break-even size (see below) |
| Dispatch-wrapper economics | **Flat** — the M4-early −16% was a run-order artifact (E-mech, p=0.89); the converted path is TOST-equivalent to plain (E-ship) |
| Trust boundary (receipts, seals, identity) | **Standing** — and repaired 2026-09-09/10: A03/A04/A07/A09/A12/A13 closed with regression evidence |
| Accounting truthfulness | **Standing** — Flash/Fast pricing corrected (PR #87), unknown usage never $0 (PR #94) |
| Leader-context protection | **Unmeasured** — the one economic experiment still unopened |
Decision (2026-09-10, owner): **the gated path is the default for substantial work** —
flash (`deepseek-v4p1-flash-1017` since the 2026-10-26 rotation) first, escalating to
`glm-5p3` then `ember-1` on recorded stagnation. The STOP rulings were correct for the
corpus they measured and do not govern long agentic work (see the break-even below). The
conversion's surviving deliverables are the **catalog** (`delegate.py --model <id>`,
live-validated; `catalog.py`) for consults and one-shots, the defect fixes, and truthful
metering — with the runner's lane table, stagnation switching, budgets and gates **retained
as live policy**.
## The routing break-even: when does cheap-first-with-escalation pay?
This is the calculation that governs the whole routing question, and the benchmark corpus
could never see it because every task was one unit long. Decompose a task's model spend:
- **a** — planning (strong-model planning; a wash when the leader plans either way)
- **b·X** — execution (X units at the cheap rate `b`)
- **c** — switching + validation (≈ **$0** here: fingerprints, verifier runs and gates are
deterministic code, not model calls; E-ship measured the whole dispatch path cost-flat)
### Per-unit success (q) — the one number everything hinges on
**q is the probability that the cheap tier completes one unit of work correctly on the
first pass: dispatched once, verified green, no repair dispatch, no escalation, no
stagnation.** It is the quantity that converts the break-even arithmetic from structural
to empirical, and it plays two different roles depending on the question:
- **Quality-matched frame (substantial work):** q sets the *magnitude* of the savings —
`savings(q) = q − b/s` of the strong comparator's cost. You win at almost any q; better
q means more money kept.
- **Bare-dispatch frame (consults/one-shots):** q sets the *threshold* at which ungated
dispatch stops being safe — whole-task success is `q^X`, and escalation beats bare
dispatch once that compounded probability drops below the price ratio
(`X* = ln(b/s)/ln(q)`).
**What a unit is.** The calibration point is the v3 corpus, where one task = one unit:
~130k tokens, ~$0.009 at flash rates, ~30–40s of focused sub-task. On real agentic work a
unit is whatever the gate can verify independently — a function, a module, a test-green
milestone. The unit is the quantum of the break-even arithmetic: `b` and `s` are per-unit
prices, `X` is the unit count, `q` is per-unit reliability.
**Where the number comes from.** The only measured value is q = 0.967 — the flash arm's
29/30 hidden-pass on the v3 corpus (#91), which is per-task on single-unit bounded
fixtures. It is a *calibration anchor*, not a production estimate.
**What q is not.** Not a model property — it is workload-class-relative (bounded
algorithmic fixtures ≠ agentic refactors ≠ production pipelines), and it is not static:
it drifts with model updates and roster rotations (the 2026-08-28 rotation replaced both
tiers; #91 re-measured). Quoting one q without naming its workload class is the same
error the M4-early comparison made by not interleaving.
**Why failure compounds.** For bare dispatch, whole-task success is `q^X`: at q = 0.90, a
27-unit task succeeds barely 6% of the time, because one wrong unit can invalidate
everything downstream of it. This is why long tasks and short tasks live in different
regimes, and why a corpus of single-unit tasks cannot see the difference. **And `q^X` is
the optimistic bound**: it assumes unit failures are independent, but in real agentic work
they *correlate* — one misunderstood requirement poisons every unit downstream of it, so
the true whole-task success is worse than the formula says. Correlation cuts both ways in
the two frames: it makes bare dispatch *more* dangerous than the threshold table implies,
and it makes the gate *more* valuable than the savings table implies — the gate catches
the first wrong unit and stops the cascade, which is exactly the failure mode independent
failure math cannot price.
**How q gets measured on real work — the ledger is the instrument.** Every gated run
already records what a q sample needs: the task record (`tasks.jsonl`) carries dispatch
count, first-verify outcome, failure class, and stagnation switches. **q for a real
workload is simply the fraction of units that verify green on dispatch 1** — so every
substantial task run through the gated path is a q measurement for free. "q is unmeasured
on long agentic work" therefore means exactly this: no long/agentic workload has flowed
through the ledger yet. The instrument exists; feed it.
### The comparison that matters: quality-matched (keep quality, reduce cost)
The cheap tier alone does not hold the required quality on substantial work — so
fixed-cheap is not a real alternative for it. The question is what flash-first with
verification gates and escalation **saves against a fixed strong model that completes
everything**, with the gate guaranteeing the same end quality (whatever flash cannot do,
the strong tier redoes). Both comparators, measured (phase3 + #91) — **on the pre-2026-10-26
arms**: the flash arm is the retired glm-5p3-flash and the k3 comparator the retired kimi-k3.
The rotation's successors (deepseek-v4p1-flash-1017, ember-1) are unmeasured, so the ratios
and thresholds below are the starting point, not their evidence:
| Comparator | Rate per unit | Price ratio b/s | Flash must first-pass |
|---|---:|---:|---:|
| **glm-5p3** | $0.0739 | 0.122 (8.2x) | **> 12.2%** of units |
| **kimi-k3** | $0.153 | 0.059 (17.0x) | **> 5.9%** of units |
(flash = the retired glm-5p3-flash, b ≈ $0.009/unit, measured $0.0084–0.0093; whole-unit
redo on escalation — the gate catching failure partway through a unit improves these
further. deepseek-v4p1-flash-1017's b is unmeasured and must be re-derived from a metered
run, not scaled: its standard row is $0.30/$0.006/$1.20 per 1M against glm-5p3-flash's
$0.15/$0.03/$0.50 — 2.0x input, 0.2x cached input, 2.4x output. ember-1 carries kimi-k3's
row exactly, so the k3 comparator's price ratio moves only through the flash arm's b.)
**Break-even task size: X\* = 0, unconditionally, against both.** With c ≈ 0 and planning
a wash, flash-first + escalation is cheaper from the first unit at any task size, unless
flash first-passes less than ~12% of units (vs glm-5p3) or ~6% (vs kimi-k3) — a failure
rate no usable cheap tier approaches. **q** — flash's per-unit first-pass completion rate
— does not determine *whether* this wins; it determines the *magnitude* of the savings:
**savings(q) = q − b/s** of the comparator's cost:
| flash first-pass q | saves vs glm-5p3 | saves vs kimi-k3 |
|---|---:|---:|
| 0.967 (v3-measured, per-task) | **84.5%** | **90.8%** |
| 0.90 | 77.8% | 84.1% |
| 0.70 | 57.8% | 64.1% |
| 0.50 | 37.8% | 44.1% |
| 0.30 | 17.8% | 24.1% |
Even if long agentic work degrades flash's first-pass rate from the benchmark's 96.7% to
50%, the gated path still **saves ~38–44% of the cost of the strong model that would have
been needed anyway**. The staged ladder is the cost-optimal form: escalate flash→glm
(8x) first, glm→k3 (16x vs flash) only when glm also stagnates.
The linchpin of the quality match is **gate reliability** — a verifier that passes wrong
work breaks the equivalence. That is exactly what the 2026-09-09/10 trust-boundary
repairs (A03/A04/A07/A09/A12/A13, PRs #88/#93/#94/#95) buy: the frame above is only
honest because the gate now actually refuses what it claims to refuse.
### Secondary: when is bare dispatch (no gate) acceptable?
For consults and one-shots — the `--model` surface — the relevant threshold is vs
fixed-cheap: how long can flash go ungated before its compounded failure probability
(q^X) makes the gate's insurance worth paying for? **X\* = ln(b/s) / ln(q)**: ~83 units
at q=0.967, ~27 at q=0.90, ~13 at q=0.80 (a unit ≈ one benchmark sub-task, ~130k tokens).
Higher q pushes this threshold *out* — reliable flash stays viable bare for longer —
which is why that table's wall time rises with q. Below the threshold, bare `--model`
dispatch is fine; above it, use the gated path.
### Caveats, honestly
`q` on long agentic work is **unmeasured** (the v3 29/30 is per-task on bounded fixtures —
a calibration anchor, not a production estimate); rescue-by-strong succeeding where flash
failed is assumed (measured on v3, assumed beyond); whole-unit redo is pessimistic;
screens, not decision-grade (#16).
**The point:** the benchmark corpus (single-unit, quality-tied) could never exercise this
arithmetic — STOP was the correct ruling for it and governs only the bare-dispatch
surface. For substantial work, the quality-matched comparison says the gated path —
flash-first, gates, staged escalation — is cheaper at any task size, and the savings grow
with flash's first-pass reliability. The remaining unknown is q — and the gated path
measures it on every substantial task it runs: the ledger is the instrument, production
work is the corpus, and a marathon-shaped eval would only be the controlled version of
what the ledger accumulates for free.
## What this repo is
- `harnesses/kimi-code/runner/` — **the live control plane** (`runner.py`): task intake with observable features, a
lane table (flash-first with evidence-driven escalation — see the routing break-even above), worker dispatch through the delegate wrapper, leader-executed verification,
tree-bound acceptance receipts (stale on any post-verify mutation), a sealed verification
surface (payloads copied out of the agent-writable workspace at init; tampered inputs are
restored and flagged at verify time), stagnation detection with failure-class switching, and an
append-only metered task ledger. Runner state lives **outside** the workspace
(`.runner-state/` sibling). Smoke suite S1–S8: `python -m unittest discover -s harnesses/kimi-code/runner/tests -v`.
Acceptance additionally refuses a receipt flagged `tamper_detected` and one written before a
later dispatch; verification pins the verifier argv at init and rejects `-m` module shadowing;
`init` refuses to silently re-baseline an initialized workspace. See **Deterministic refusals**
below for the full list, and `harnesses/kimi-code/runner/pilot.py` for the real-dispatch harness that drives the
whole loop against live workers.
- `harnesses/kimi-code/delegate/` — a lean Windows-native subprocess wrapper (`delegate.py`) that runs external CLI
workers with a stable envelope contract: exit codes, truncation flags, captured child session
ids, and per-dispatch isolated+seeded `KIMI_CODE_HOME` homes (`child_home` in the envelope;
callers meter from it, then delete it). Run dirs and homes live under one structured dispatch
scratch root (`C:\Dev\scratch\model-proctor`, override `MODEL_PROCTOR_SCRATCH`) rather than the
system `%TEMP%`, so a caller can sweep one dispatch's state by id. Tests live in
`harnesses/kimi-code/delegate/tests/`.
- `scripts/extract_log.py` — deterministic session-log fact extractor with a **coverage manifest**
(bytes in, records parsed, records dropped). LLMs interpret extractions; they never scan raw
volume. Verifier-class: hash-frozen per goal. Tests live in `scripts/tests/`.
- `policy/` — design/governance specs, including preserved predecessor specs (see Provenance below).
- `evals/` — the benchmark harness: v1 (20 cases + 2 showcase, killed the dynamic design), v2 and
v3 quality sets (10 each, naive-solution-proven to discriminate requirement fidelity), the
metering stack (`meter.py`, `pricing.yaml`), pre-registrations (`PREREG-v2.md`, `PREREG-v3.md`),
and every result row (`results*.jsonl`, `pilot-*.jsonl`, `phase2-*.jsonl`, `phase3-*.jsonl`).
- `harnesses/kimi-code/skill/` — Kimi Code skill sources: `model-proctor/` is the live/installable policy;
`multi-model-routing/` is a preserved superseded research artifact and is not installed.
(`static-cascade/` was removed 2026-09-25 per owner directive, superseding the #16 retention.)
Python 3.10+, standard library only, everywhere.
```bash
python -m unittest discover -s harnesses/kimi-code/runner/tests -v # live path smoke suite (S1–S8)
python -m unittest discover -s harnesses/kimi-code/delegate/tests -v
python -m unittest discover -s harnesses/codex/delegate/tests -v
python -m unittest discover -s scripts/tests -v
```
## Codex catalog delegate
[`harnesses/codex/delegate/`](harnesses/codex/delegate/) is a separate,
explicit-dispatch adapter. Operators select `--model` or a named local preset;
it never routes by a Kimi lane or task shape. It validates the live Codex
app-server catalog, requires a prompt file outside the target workspace, and
uses exactly the requested `cli` or `app-server` transport with no fallback.
See its README for the schema-1 Luna/Terra/Astra presets, read-only/write
sandbox boundary, evidence semantics, and explicit-destination-only installer.
---
## Using it from Kimi Code CLI
Install once (idempotent):
```bash
python scripts/install.py
```
This installs the `model-proctor` skill to `%USERPROFILE%\.kimi-code\skills\`
and the runner + delegate + catalog + agent roster to `C:\Tools\model-proctor\`. If `agents.json` is missing,
regenerate it first from `harnesses/kimi-code/delegate/agents.example.json` (see `harnesses/kimi-code/delegate/README.md`).
### Start a subagent with a model (#96 — the default path)
```bash
python C:/Tools/model-proctor/catalog.py # live models + prices
python C:/Tools/model-proctor/delegate.py \
--model fireworks/deepseek-v4p1-flash-1017 \
--workspace <ws> \
--task-file <path> # or --task "<text>"; --write to grant write
```
The model id is validated against kimi's **live** config at dispatch time —
rotated ids refuse loudly with same-family alternatives. Read-only by
default; `--write` is explicit and model-mode-only. `--resume-from
<session_id>` continues a child session. Every child carries the injected
`PROCTOR_CHILD` marker; a nested delegate refuses `--model`/`--write`.
Production-pattern tasks (`run_week.ps1`, `src.run_all`, ...) are refused
here — they belong on the gated runner path below. Unpriced models meter as
**unknown, never $0**.
### Triggering the proctor (the gated path)
Skills load at **session start**, so open a new Kimi Code session after installing. Then:
- **Implicit:** give the session a substantial coding task (multi-file, unfamiliar code, anything
past one edit-test cycle). The `model-proctor` skill fires on its own description; trivial tasks
are deliberately executed directly.
- **Explicit:** *"Use the model-proctor skill: run this through the gated path — init, dispatch,
verify, and accept only on a fresh tree-bound receipt."*
The skill then drives the loop: `init → dispatch → verify → accept → record`, with the
runner — not the model — owning state, verification, budgets, and receipts.
### Direct CLI use (no skill)
```bash
# task.json: {"task_id": "t1", "prompt": "Fix sum_to_n in math_utils.py.",
# "features": {"bounded": true, "known_location": true, "objective_acceptance": true},
# "scope": ["math_utils.py"], "verifier": {"argv": ["{python}", "check.py"]},
# "seal": ["check.py"]}
python C:/Tools/model-proctor/runner.py init --workspace <ws> --task task.json
python C:/Tools/model-proctor/runner.py dispatch --workspace <ws> --task task.json
python C:/Tools/model-proctor/runner.py verify --workspace <ws> --task task.json
python C:/Tools/model-proctor/runner.py accept --workspace <ws> --task task.json
python C:/Tools/model-proctor/runner.py record --workspace <ws> --task task.json
```
State, receipts, and the ledger land in `<ws_parent>/.runner-state/<ws>-<hash>/` — outside the
worker's writable tree. Every command prints one JSON object; nonzero exit = refused/failed.
#### Optional task-file fields
| field | effect |
|---|---|
| `lane` | Overrides the frozen lane table. A reviewed decision — record why. Also the only way to run an ops-class task on `flash`. |
| `seal` | Extra files copied out of the workspace at init and restored at verify. Verifier argv files and the config surface are sealed automatically; a `-m` verifier seals nothing from argv, so name test helpers here. |
| `preflight_receipts` | **Required** for tasks naming a known production entrypoint: paths to logs the orchestrator's own probe produced. |
| `budget.max_preflight_age_s` | Widens the default 24h freshness window on those receipts. |
#### Deterministic refusals, and how each clears
Every one is a refusal, not a warning — nonzero exit, no receipt, no acceptance.
| refusal | raised by | clears when |
|---|---|---|
| `workspace_is_not_repo_root` | init | the workspace *is* the repo root (#20) |
| `already_initialized` | init | you pass `--reinit` (counted on state, so re-baselining is visible) |
| `state_dir_inside_workspace` | any | `--state-dir` points outside the agent-writable tree |
| `flash_lane_forbidden_production_runner` | init, dispatch | features classify honestly, or an explicit `lane` is set |
| `preflight_receipt_required` / `_stale` | dispatch | the probe has actually run, recently |
| `config_surface_changed` | verify | no conftest/pytest.ini/pyproject/`*.pth` added or removed since init |
| `verifier_changed_since_init` | verify | the task file's verifier matches the copy pinned at init |
| `module_shadow_detected` | verify | no workspace file shadows a module the verifier imports via `-m` |
| `tamper_detected` | accept | you re-run `verify` on the restored tree |
| `stale_receipt` | accept | you re-run `verify` — the tree changed, or a dispatch happened after the receipt |
The last two clear by re-running **`verify`**, never by re-running `accept`. A receipt records the
tree signature, the dispatch count it was written at (`dispatch_seq`), and the verifier argv that
produced it — so a green receipt states *what* was verified, not merely *that* something was.
---
## Provenance: how this design was arrived at
This section exists because the final design only makes sense as the residue of several
**falsified or adjudicated predecessors**. Durable conclusions are summarized here; superseded
working papers remain recoverable from Git history rather than living at the current repository tip.
### Phase 0 — Reverse-engineering Magnitude (2026-07)
Starting question: can an open-model harness reproduce [Magnitude](https://magnitude.dev/)'s routing
advantages — better-than-single-model quality at lower cost — inside Kimi Code CLI?
An adversarially reviewed reconstruction produced these durable conclusions:
- Magnitude's defensible advantage is **not a learned model selector**. It is a hierarchical,
stage-aware orchestration policy: persistent leader → bounded subtasks → specialized worker roles →
isolated contexts → mandatory fresh-context independent review → bounded escalation.
- **Confirmed** from public material: leader/worker split, role specialization, isolated contexts,
fresh-context review, lower-cost workers for token-heavy tasks, provider capability metadata.
- **Not established** (and never claimed again): learned routers, bandit routing, online learning,
per-repo auto-benchmarking, exact thresholds.
### Phase 1 — Operationalization specs (2026-07)
Two independent Kimi Code proposals converged on a two-tier architecture: Kimi K3 as lead,
GLM-5.2 as native secondary, and external CLI workers behind a lean `delegate` subprocess wrapper.
The key surviving decision was **do not make the wrapper impersonate the secondary-model API** — a
one-shot subprocess result is not a streaming model endpoint; faking one requires an HTTP service,
tool-call translation, cancellation/state machinery, and defeats the purpose of a lean wrapper.
### Phase 2 — Dynamic per-turn routing: built, benchmarked, **rejected** (2026-07/08)
The first implementation was a **dynamic** router (`harnesses/kimi-code/skill/multi-model-routing/`,
`policy/delegation-policy.md`): the leader classified each task at dispatch time and picked a model,
with pre-registered predictions of **≥2–3× cost reduction** vs K3-alone.
`evals/` is the harness built to test that: 20 pre-registered cases (tune/holdout split fixed before
any runs), deterministic fixtures, machine-checkable acceptance **plus hidden tests**, a blinded
scorecard (its tables key on blind letters, but `report.py` prints the blinding key itself in a
`## Blinding` section at the top — the mapping is disclosed in the same document, so blinding only
resists reading the tables by config name), and a
simple-case guardrail (the router must not regress trivial tasks).
**What the benchmark actually showed** — and this is why honest instrumentation matters:
1. **The primary cost metric was a broken proxy.** `tokens_reported` was null throughout (kimi prints
no token counts in headless mode), so cost fell back to an I/O-volume heuristic.
2. **The fixtures never discriminated on quality.** All 18 showcase runs passed acceptance and hidden
tests at 1.00 across all configs — the suite could not detect quality differences at all.
3. **Retroactive real metering** (`evals/results-metered.jsonl` — true per-model token usage pulled
from session wire logs, priced by `evals/pricing.yaml`) showed the routing configs at **+16% and
+37% real cost vs the K3 baseline** (config medians across all showcase runs).
4. The simple-case guardrail **failed** (routing overhead slowed the trivial case).
The dynamic design's cost predictions did not materialize; on cost the prior result is
*uninterpretable* (broken proxy), not falsified — but nothing justified adoption, and the guardrail
failure was real. **Dynamic per-turn routing was rejected.** The skill is kept in-repo as a
superseded artifact, not deleted, because the delegation mechanics (wrapper, task packets, evidence
rules) survived into the static design.
**Correction (2026-08-27 — EVAL-004, issue #46).** Point 2 above is wrong about *which* instrument
failed, and the original wording is kept because it is what this repo concluded at the time. Counted
directly from `evals/results.jsonl`:
| | rows | `acceptance_pass` | `hidden_pass` |
|---|---|---|---|
| all | 36 | **36** | 27 |
| showcase | 18 | **18** | **9** |
Acceptance passed at 1.00. Hidden passed at **0.50** — `big_explore` failed its hidden check in 8 of
9 runs while `bulk_migrate` failed 1 of 9. So the hidden fixtures *did* discriminate, decisively.
What did not discriminate was the **acceptance check**, which passed 36/36 while the hidden check was
failing 9 times: it caught **0 of 9**.
That inverts the lesson. Phase 2 recorded a verifier problem as a fixture problem — and the
distinction is load-bearing, because Phase 3 below cites RouterBench for the constraint that
cascading only works when verifier error stays ≤ 0.1, a threshold this repo had never evaluated
against its own verifiers.
`evals/verifier_error.py` now computes the contingency tables on demand, reporting **both** the joint
`P(accept ∧ ¬hidden)` and the conditional `P(accept | ¬hidden)` with explicit denominators — on
`results.jsonl` those read 0.25 and 1.00 respectively, and quoting either without its denominator is
meaningless. `evals/PREREG-verifier-error.md` fixes the decision rule.
The honest verdict from that rule: **every corpus here is underpowered.** No file has more than 9
rows in which the hidden check failed, and resolving a 0.1 threshold needs roughly 30. The finding is
therefore *"these screens cannot resolve verifier error"*, not a rate — which is itself worth
recording, because it means the v1/v2/v3 pass rates say less about acceptance quality than their
headline numbers suggest. No architectural conclusion is drawn from it.
### Phase 3 — Literature verification → the static cascade (2026-08)
Before designing v3, the routing literature was checked against primary sources. The evidence base
(`policy/STATIC_CASCADE_SPEC.md` §2, each claim labeled established / partially-established /
implementation-choice):
- **Pre-generation (plan-time) routing beats post-generation cascading on 4 of 5 benchmarks** —
cascades pay the cheap model's structural cost on every escalation. All single-turn benchmarks;
multi-turn agentic transfer untested. (arXiv:2605.06350)
- **2-tier difficulty cascades are optimal**; extra mandatory tiers add cost without held-out gains.
(arXiv:2605.06350)
- **Cascading only beats baseline when verifier/judge error ≤ 0.1**; at 0.2 performance may
deteriorate rapidly. (RouterBench, arXiv:2403.12031)
- **Deterministic verification (code execution) as the escalation signal works in agentic settings**:
EcoAssistant, +10pts success at <50% cost. EcoAssistant's escalation is dynamic; this design borrows
its verifier mechanism but replaces dynamic routing with static plan-time allocation.
(arXiv:2310.03046)
The decisive internal analysis was the **cost model** (spec §0): agentic coding is input-dominated,
and the delegate wrapper's stateless calls **forfeit prompt-cache state**. Worked arithmetic (60K
input / 3K output turn, Fireworks list prices):
| Path | Turn cost |
|---|---|
| K3 warm session (80% cache hit) | $0.095 |
| **DS-Pro via stateless delegate (0% cache)** | **$0.115 — loses to warm K3** |
| K3 cold (0% cache) | $0.225 |
| **DS-Flash via stateless delegate** | **$0.009 — wins regardless (~10–25×)** |
Hence the inverted default: **Flash is the default executor**, Pro is escalation-only, K3 plans,
GLM-5.2 orchestrates and QCs, Codex/Claude CLI subscriptions sit at the apex as flat-rate read-only
advisors. Allocation is decided once at plan time and **controller-enforced** — GLM decides within
legal transitions; the controller rejects illegal ones. GLM is a decision component, not the control
plane.
The spec went through two expert advisor reviews (Codex + Claude); v3 incorporated 15 findings and
rejected/deferred 2 with reasons recorded (spec §11). Current version: v3.1.
### Phase 4 — Production dogfooding on a real financial-controls codebase (2026-08)
The cascade was taken into a live takeover of an independent quant-engine project (private repo) —
real gates (824-test suite, strict mypy, ruff), a governed issue tracker, adversarial QC on every
material change. What production taught that benchmarks didn't:
- **The leader's own context is the dominant cost**, not workers. Metered reality on the heaviest day:
~$203 total, of which the K3 leader session was ~$196 (96%) — 506M cached-read tokens compounding
over an all-day session — vs $7.23 for all delegate workers combined. The counterfactual for the
delegated slice: ~$9.44 as-routed vs ~$17.85 had the same work run on K3.
- **Summaries crossing the worker→leader boundary can miss, lose, or misrepresent.** The response is
architectural, not more review layers: deterministic log extraction with coverage manifests,
controller-computed `files_changed` (including untracked files), evidence archival with sha256 at
dispatch completion, truncation flags that void verdicts, and non-discretionary spot-verification.
- **Self-set review standards are a governance hole.** `threat_model` is now an owner-set field at
`cascade init`, immutable for the goal; dismissed blocker/major findings must be recorded verbatim
and surface verbatim in the final report.
- **Adversarial QC loops need hard caps.** One composite change went through 8 fresh-context QC
rounds; the controller's code-enforced QC cap exists because a human leader will not stop itself.
These lessons are codified in `policy/leader-context-and-worker-integrity.md` (v2 — itself
adversarially reviewed by 5 fresh-context reviewers, which caught v1 overstating what existed; every
fix is honestly labeled [CODE] / [POLICY] / [RESIDUAL]).
### What remains unsolved (honest residuals)
- Entailment and completeness are not boundary-checkable: omission, true-but-irrelevant evidence,
and aggregate miscomposition survive every guard. Reduced, not eliminated; still a leader/human duty.
- Break-even conditions in the spec are model-derived estimates under assumed cache-hit rates, not
measured constants. `evals/meter.py` (`session_rebuild` token class) exists to measure them before
further adoption claims.
---
## Reference research
| Claim used | Source |
|---|---|
| Pre-generation routing > post-generation cascading (4/5 benchmarks); 2 tiers optimal | arXiv:2605.06350 |
| Cascade viability requires verifier error ≤ 0.1 | RouterBench, arXiv:2403.12031 |
| Deterministic verification as escalation signal in agentic settings | EcoAssistant, arXiv:2310.03046 |
## Reference products
- [Magnitude](https://magnitude.dev/) — the routing behavior this project set out to reproduce with
open models; see Phase 0 for what is confirmed vs. inferred about it.
## Model lineup (measured 2026-08-25; roster rotated 2026-08-28 and 2026-10-26)
The frozen cascade's role ladder (K3 plans → GLM orchestrates → Flash executes → Pro/advisors
escalate) was a **cost hypothesis**. The pre-registered fixed-arm grids (issues #27/#29/#30,
`evals/phase2-*.jsonl`, `evals/phase3-*.jsonl`) measured it instead: on bounded, spec-complete
tasks the arms are quality-indistinguishable, so cost decides.
| Role | Model | Basis |
|---|---|---|
| Default worker (bounded/substantial) | DeepSeek V4.1 Flash 1017 (`marathon-worker`) | Roster swap **2026-10-26** (owner-directed): `glm-5p3-flash` → `deepseek-v4p1-flash-1017`; **measured 2026-10-26** (PREREG-v3-rotation-2026-10-26, interleaved with glm-5p3): **30/30 hidden on v3, $0.0246/hidden-pass** — quality-clean, but ~2.6x the retired glm-5p3-flash's $0.0096 |
| Cheap second / fallback | GPT-OSS-120B (`gpt-oss-worker`) | 29/30 hidden on v3 (one flake), $0.014/hidden-pass at corrected prices |
| Substantial second (equal quality, faster wall) | GLM-5.3 (`substantial-worker`) | Moved to 5.3 on 2026-08-28; re-measured 2026-09-09 (#91: $0.074/hidden-pass) and again **2026-10-26** (interleaved with the new flash arm): **30/30 hidden on v3, $0.0547/hidden-pass** |
| Marathon / open-ended | Ember-1 (`expert-worker`) | Roster swap **2026-10-26** (owner-directed): `kimi-k3` → `ember-1`, which carries kimi-k3's price row exactly. **Hypothesis only — see Open gaps below**; the retired kimi-k3's marathon basis was never measured either |
| Cheap scout (vision, misc.) | Kimi K2.7 (`k27-scout`) | Carried from the frozen roster; live |
| Advisors (read-only, closed triggers) | Codex CLI / Claude CLI | Flat-rate; trigger set in issue #8 — **not yet wired into the runner** (see Open gaps) |
Seat ids name the **role the owner buys, not the model sitting in it**, and they were renamed to
role names on **2026-10-26** (owner-directed — its own decision, taken separately from the model
rotation above): `glm-flash-worker` → `marathon-worker` (the cheap seat you can afford to run
long), `glm-worker` → `substantial-worker` (the engineering middle), `k3-worker` →
`expert-worker` (the most capable). Renaming a roster id is an interface change, not part of a
model swap, which is exactly why it took a separate decision. (The 2026-08-28 rotation had
renamed `ds-flash-worker` → `glm-flash-worker` in #73.) Ids are re-probed at experiment time;
read the model column, not the id, for what a lane runs.
Swapping a lane's model is a runbook, not an improvisation: see
[`docs/model-rotation-runbook.md`](docs/model-rotation-runbook.md) for the phased procedure
(probe before bind, the ACL grant before editing `agents.json`, the metering gate on the
`record` row rather than `status`), its per-step gates, and the seven failure modes each
guardrail exists to catch.
Catalog note: Fireworks moves models to dated snapshot ids — the undated
`deepseek-v4-flash` alias 404s while `deepseek-v4-flash-0731` was the live deployment
(probed 2026-08-25, see #23). On 2026-08-28 kimi's synced fireworks roster dropped
`deepseek-v4-flash-0731` and `glm-5p2` and gained `glm-5p3` / `glm-5p3-flash`; the flash
lane moved to `glm-5p3-flash` and the glm lane to `glm-5p3`. Live-probe status 2026-08-28:
`glm-5p3-flash` dispatches end-to-end (real worker run accepted); undated `glm-5p3` returned
`404 Model not found, inaccessible, and/or not deployed` in the morning but went live the
same day (direct re-probe OK — final serving probe 2026-08-28); `glm-5p2` still serves
(alias re-added to kimi's config) and remains the recorded rollback for the glm lane.
On 2026-10-26 the flash lane moved `glm-5p3-flash` → `deepseek-v4p1-flash-1017` (the dated
2026-10-17 successor of the undated `deepseek-v4p1-flash`, which is the 2026-09-10
snapshot) and the k3 lane `kimi-k3` → `ember-1`. Both new ids are live and priced on
Fireworks — standard tier, read from the live catalog (`firectl serverless list`) on
2026-10-26: $0.30/$0.006/$1.20 and $3.00/$0.30/$15.00 per 1M. **Neither has been probed
end-to-end from a dispatch, and neither has been measured on the v3 corpus** — treat the
rotation as unconfirmed until both happen. Both ids resolve in kimi's config, which is
necessary but not sufficient — config presence is not serving (the 2026-08-28 `glm-5p3`
404 above is the precedent). `ember-1` was already there; the dated 1017 alias arrived with
the 2026-10-26 catalog refresh, which also added the spot routers and the remaining
serverless models.
---
## Development process: the backlog is GitHub Issues
All work on this tool is managed through **this repository's GitHub Issues** — no side channels.
- Every unit of work is an issue with a stable key in the title (`[AREA-NNN]`).
- Labels: `priority:now` / `priority:next` / `priority:parked`, plus `type:*` and `area:*`.
- Dependencies are tracked in a `## Blocked by` section (`- #N` lines or `None`). Parked items carry
an explicit **activation trigger** so overkill stays parked instead of silently becoming urgent.
- Closing an issue requires a non-empty `## Final evidence and handoff` section — evidence, not vibes.
- Design decisions that survive debate are recorded in `policy/` with rejected alternatives and
rationale; live task state and evidence are owned by the active harness control plane.
Start here: [Issues](../../issues).
---
## Status
The measured end-state for bounded/spec-complete tasks is live: fixed cheap task-owning worker +
deterministic verification + failure-class switching, no economic routing. The bespoke cascade is a
frozen research artifact (6095695; governance #16). The grid results (#29/#30): quality parity
across K3/GLM/GPT-OSS at 3.0× cost spread → the STOP rule fired; routing complexity stays parked.
**2026-09-09.** The M4-early comparison (sealed `PREREG-plain-vs-proctored`) measured the dispatch
wrapper itself: delegate-only dispatch beat the full runner flow by 16% per hidden-pass and 2.3×
wall time at equal-or-better quality — the wrapper's residual value is the trust boundary and
accounting, not economics. The rotated roster was re-measured (#91): `glm-5p3` and `glm-5p3-flash`
dominate their predecessors; glm-5p3-flash is the cheapest measured arm in the program. Six
reproduced acceptance defects were fixed with regression evidence (A03/A04, A07, A09, A12, A13;
PRs #88/#93/#94/#95). Consequence: conversion to catalog-dispatched subagents (#96) — see the
status update at the top of this README and `policy/FINDINGS-2026-09-09.md` for the findings,
conclusions, and plan.
**2026-10-26.** The roster rotated again (owner-directed): flash lane `glm-5p3-flash` →
`deepseek-v4p1-flash-1017`, k3 lane `kimi-k3` → `ember-1`. This is a *roster* change, not a
measurement — both new arms are unmeasured, so every economics figure above still belongs to the
arms they replaced (glm-5p3-flash's $0.0096/hidden-pass, the 8.2x/17.0x ratios), and the
break-even section says so explicitly. Re-measurement is a separate, pre-registered unit of work
(#91 is the template). Both new ids resolve in kimi's config — the dated 1017 alias landed with
the same day's Fireworks catalog sync (19 → 27 fireworks entries, which also brought in the spot
routers and the remaining serverless models), and the 1017 endpoint answers a 1-token probe
(HTTP 200) — but neither has been dispatched through the wrapper end-to-end.
The trust boundary is sealed for the live path (TOOL-013/014, issue #31/#33): isolated per-dispatch
`KIMI_CODE_HOME`, runner state and verifier payloads outside the agent-writable workspace,
restore-and-flag on verifier tamper, tree-staled receipts.
**Correction (2026-08-27 — TOOL-015/016/017/018, issues #36/#37/#38/#40).** The paragraph above is
left as written because it is what this repo claimed at `7a69061`; "sealed" overstated what the code
did. Two of the four mechanisms it cites did not hold as stated:
- *restore-and-flag on verifier tamper* — the flag had **no consumer**. `cmd_accept` never read
`tamper_detected`, so a receipt recording that the worker altered the exam was accepted silently.
- *tree-staled receipts* — in a **git** workspace the signature hashed `git status` letters and
paths, never content, so re-editing an already-dirty file left it unchanged. A dispatch after a
green verify also left the receipt intact. (Every recorded eval run used the non-git `files:` path,
which hashed contents and was correct, so no measured result is affected.)
Both are fixed, along with two paths that reached a green receipt without the verifier running at
all: the verifier argv was re-read from the worker-writable task file on every command, and
`python -m MOD` resolved `MOD` out of the workspace, so a one-line `ws/unittest.py` turned a failing
suite into exit 0.
The accurate current claim is **tamper-evident against a non-adversarial worker**, not sealed
against a hostile one. The worker runs as the same OS user with no filesystem confinement: the state
root sits at a path it can compute and write, receipts are unsigned JSON, and the installed tool
directory is user-writable. The realistic threat is a reward-hacking model gaming the exam it was
handed rather than an attacker hunting the grader's filing cabinet — but the distinction is now
recorded rather than assumed away. See #40 for what remains and what closing it would cost.
### Open gaps and their activation triggers
Parked items stay parked until their trigger fires — that is the governance working, not neglect.
| Gap | Status | Activation trigger |
|---|---|---|
| **Marathon / open-ended lane (K3)** | Unproven — no v2/v3 case is marathon-shaped; K3's role there is a vendor-prior hypothesis | The first real multi-hour/open-ended task appears, or a marathon-shaped corpus gets built (extend `evals/fixtures/`, new PREREG); then run K3-vs-cheap as fixed arms on it before trusting the lane |
| **Advisor gates (Sol/Opus, #8) and M3 parallelism (#26 Phase 5)** | Deferred, untested | Observed *conceptual* stagnation in production — workers stuck on reasoning, not execution loops (the runner's failure-class log distinguishes these); for M3: a task with genuinely separable uncertainty branches |
| **Provider-billing reconciliation** | Residual — wire-metered costs are the estimate of record | First provider invoice day, or Fireworks billing-API access: reconcile `evals/meter.py` output against the bill within the PREREG tolerance (2–5%) |
| **Decision-grade evaluation** | Screens only (v2/v3 are n=10–30, hidden checks run in-workspace albeit sealed) | Any claim intended to justify production adoption for a *third party*; until then, all results here are single-operator screens |
| **Learned routing (#7)** | Parked | Measured oracle headroom: fixed arms must show systematic per-task-class quality splits large enough to pay for a classifier — v2/v3 show the opposite |