You close the harness improvement loop: execute → trace → diagnose layer → propose
minimal edit → regression validate → promote or reject. Model weights are out of scope.
Hard Rules
Never run an evolution round without harness vN manifest and operational eval harness.
Never propose an edit without trace evidence tied to a failure cluster.
Never accept an edit without regression gate: held-in Δ ≥ 0 AND held-out Δ ≥ 0 AND max(Δ) > 0 (Self-Harness).
Never make outcome-only edits — attribute failure to an ETCLOVG layer first (HarnessFix).
Never allow evolve agent to modify verifier config, eval held-out tasks, or LLM API keys (AHE sandbox).
Never promote prompt-only changes when tools/middleware/skills are the diagnosed layer (AHE ablation).
Never bypass file-scope guard — edits only to paths declared in manifest allowed_write.
docs/harness/eval-interface.md + regression task set defined (else → eval-rubric-design → eval-pipeline).
Held-out split documented — never fed to proposer (Self-Harness, Meta-Harness).
FAIL fast with specific route if any missing.
Step 1 — Capture traces
Collect from: benchmark runs, docs/memory/agent-handoffs.md, session logs, or
docs/harness/runs/iteration_NNN/. Distill to layered digest per AHE experience observability —
raw millions of tokens are not fed to the proposer.
Step 2 — Diagnose (ETCLOVG + HTIR)
Per references/diagnosis-etclovg.md:
Normalize traces to step-level nodes (HarnessFix HTIR pattern).
Attribute each failure cluster to one primary layer: Execution, Tooling, Context,
Lifecycle, Observability, Verification, Governance.
Consolidate recurring flaws into actionable records — one mechanism per record.
Step 3 — Propose diverse-minimal candidates
Generate K candidate edits (default K=3), each:
Tied to one failure mechanism (Self-Harness).
Scoped to manifest allowed_write paths (metaharness scope guard).
Write docs/harness/evolve/change_manifest.json before evaluation.
Step 4 — Regression validate
Invoke eval-pipeline (harness regression mode) on held-in + held-out splits.
Gate
Rule
Self-Harness acceptance
held-in Δ ≥ 0 AND held-out Δ ≥ 0 AND improvement > 0
Scope
No files outside allowed_write
No-change
Zero file changes → inherit parent scores, do not promote (metaharness)
pass@1
Optimize pass@1, not pass@k flaky strategies (AHE)
Step 5 — Promote or reject
Accept: bump manifest version to vN+1, update hashes, archive run under docs/harness/runs/.
Reject: log predicted-vs-actual in manifest; if same flaw persists 2+ rounds at same layer → rollback component and pivot layer (AHE).
Optional label-free path (RHO): when no labeled eval exists, use self-consistency +
pairwise self-preference among candidates — still require positive mean score before promote.
Step 6 — Memory + handoff
On promote: memory-capture with harness version, delta metrics, and changed components.
Append docs/skill-outputs/SKILL-OUTPUTS.md.
Gotchas
Compressed feedback loses credit assignment — never reduce traces to scalar score only (Meta-Harness).
Runtime supervision patches suppress errors without fixing harness flaws — reject as edits (HarnessFix).
Self-attribution misses regressions — manifest must list risk_tasks predicted to break (AHE).
Generic prompt bloat — every instruction must map to a diagnosed failure cluster.
Label-free RHO is fallback — prefer verifier-backed regression when labels exist.
