calibrate

# Calibrate a harness You are the outer loop. The inner harness is any harness on this platform: a System One reflex over an environment, or a System Two agent over a task suite. You change its **configuration** (what its model is told, shown, allowed, and how often it acts) and nothing else: never the model, never the environment's truth, never anything that chooses in the model's place. Everything you need is in the inner harness's package: `config.yaml` (the configuration, versioned; `objective` says what counts as success, failure, ordered metrics, locus and evidence) and `ledger.jsonl` (every version so far, with its evidence and verdict). Fetch it first with `scripts/fetch.py --out package` (the package that carries the environment), then read both. Never use the harness's plugin export for this: it is a generated package with no version, and publishing it back adds a second package beside the real one. The harness's id, the platform URL and your credential are in `HR_INNER_HARNESS`, `HR_API_URL` and `HR_CALIBRATION_TOKEN`. The scripts under `scripts/` do the platform work; read `--help`. ## The method 1. **Baseline.** `scripts/bench.py --runs 3` starts three runs, one at a time (never two on one machine: shared machines drop frames and fake regressions), fetches each run's `trace.json` and its evidence archive, and prints the objective's scoreboard and the failures grouped by locus. Write the scoreboard down. 2. **Read.** Take the largest failure group. Render its evidence (the package may ship a renderer under `tools/`, for example `tools/evidence.py <trace> <observations> <out>`; read the images it writes). Read the trace around the failing steps: the state the model saw, the questions, its probabilities. Say in one sentence what killed the run there. 3. **Measure.** No change without a measurement. Probe the environment at that locus with `scripts/probe.py "action,action,..."` (a scripted run) until the mechanism is a number or a reproducible case: a distance, a window, a timing, a wrong sentence in the state. 4. **Change one thing.** Edit `config.yaml`: one fact or rule in `instructions`, one tunable, one gate threshold, or one encoder setting. One change. Bump `version`. Append a ledger line with the evidence (session ids, file paths, the measurement) and `verdict: pending`. If the state itself is wrong (the environment lied), do not patch around it: write the failing case down and stop with a proposed code fix for a person to merge. 5. **Publish and validate.** `scripts/publish.py` uploads the package as the harness's plugin and relaunches it. Run the bench again (same run count). The verdict comes from those runs and the ordered metrics only: `kept` if they improved in order, `reverted` if not. On `reverted`, put the previous version back and publish again. Record the verdict in the ledger line. 6. **Stop** at the target, at the budget, or after three reverted versions in a row. Report the scoreboard before and after, the ledger lines you added, and which limit stopped you: the world's truth, the pace, or the model. ## What never changes - One change per version. Batching four changes cost a night of untangling. - The verdict comes from the inner harness's own model runs. A stand-in that follows the rendered advice deterministically went to zero failures while the model went from three passes in three to none in six. - Read failures from the evidence, not from the summary text. - Reversible and attributed: every version is in the ledger with what it changed and why.