harness-improvement-loop

name	harness-improvement-loop
description	Use this skill after a trace-backed agent failure or repeated harness friction. Turns the failure into a ranked harness change, records a prediction, applies the smallest prevention, and reruns the relevant eval/regression gate. This is the agent-harness-kit AHE-lite loop.
allowed-tools	Read, Write, Edit, Grep, Glob, Bash(git diff:*), Bash(npm run harness:eval:*), Bash(node .harness/scripts/improvement-bundle.mjs:*), Bash(node .harness/scripts/record-failure.mjs:*), Bash(node .harness/scripts/check-failure-records.mjs:*), Bash(node .harness/scripts/harness-report.mjs:*), Bash(node .harness/scripts/regression-runner.mjs:*)

Harness Improvement Loop

When to use

• /trace-analyzer found a durable harness failure.
• A skill, hook, agent, or eval needs to change because the same mistake can recur.
• The user asks to "make the harness learn from this", "AHE-lite", or "prevent this next time".

Steps

1. Read the trace analysis and name the smallest prevention target.

2. Gather deterministic signals before deciding. Run the runtime-agnostic improvement bundle:

   node .harness/scripts/improvement-bundle.mjs --window 14
   

   Treat taxonomy_classification, suggested_records, and nextSteps as the machine-readable starting point. If no signal appears, use the user's failure report as evidence and record that evidence path explicitly.

3. Read .harness/failures/taxonomy.json. If no failure record exists yet, run the relevant suggested record-failure.mjs command from the bundle, or create one with .harness/scripts/record-failure.mjs. It writes .harness/failures/records/<id>.json, emits promotion nextSteps, and validates the record.

4. Write a prediction record before editing: .harness/improvements/<YYYYMMDD-HHMM>-<slug>.json

5. Include these fields:

   • failureClass
   • preventionTarget
   • failureRecordPath
   • expectedMetric
   • expectedDirection
   • baselineEvidence
   • verificationCommand

6. Apply the smallest change in one place: skill, hook, subagent, deterministic script, eval task, permission policy, structural rule, or docs.

7. Run the matching verification command. Prefer the narrow eval first, then a broader regression run if the change touches shared harness behavior.

8. Run .harness/scripts/check-failure-records.mjs.

9. Run .harness/scripts/harness-report.mjs --json --fail-on=fail --review-promotion=fail and follow any repairCommands or nextSteps it emits.

10. Append the observed result to the prediction record and promote the failure record with .harness/scripts/record-failure.mjs --update=<id> --status=applied|verified .... Do not hand-edit promotion state unless the helper is unavailable.

Prediction record shape

{
  "failureClass": "context-miss",
  "preventionTarget": "skill",
  "failureRecordPath": ".harness/failures/records/20260527-context-miss.json",
  "expectedMetric": "task pass rate",
  "expectedDirection": "increase",
  "baselineEvidence": ".harness/eval/results/latest.jsonl",
  "verificationCommand": "npm run harness:eval -- --quick --transport=mock",
  "observedResult": null
}

Output contract

### Harness improvement
**Failure class:** <class>
**Signal bundle:** <bundle command or evidence path>
**Changed:** <file list>
**Prediction:** <metric direction>
**Verified:** <command and result>
**Remaining risk:** <known gap or none>

Anti-patterns

• Do not change a broad prompt when a deterministic script or eval can prevent the issue.
• Do not mark the improvement successful without recording observed results.
• Do not bundle unrelated harness improvements into one loop.