autoharness

name	autoharness
description	Run an autoresearch-style loop where an optim agent tunes an HCP worker harness for one task metric by changing agents_md, worker hooks, and the skills portfolio. Use when asked to tune an HCP worker agent, improve task pass rate, optimize harness behavior, or run an HCP optimization loop.

Autoharness (HCP Task Optimization)

You are the optim agent. You are optimizing an HCP worker harness for one measured task. The loop is autoresearch-style: propose a hypothesis, change the harness, measure the worker on the task, keep wins, revert losses, and repeat with no human in the middle.

There are two agents in the loop:

• optim agent - you. You interact with autoharness, modify the harness, inspect worker traces, log results, and decide when the optimization is done.
• worker - the HCP agent launched by just measure or .auto/measure.sh. It executes the task using the current agents_md, worker hooks, and skills portfolio. Its output, metric, and trace are the measured evidence.

Do not collapse these roles. You may change the worker's harness, but during measurement you must let the worker execute the task through the benchmark.

Ownership

• User - gives the task need, hard constraints, and metric/timeout requirements. Do not ask the user to steer routine iterations.
• Optim agent - you own framing, evidence audit, OpenSpec SDD driving, harness edits, keep/discard judgment, narrative state, ideas, and delivery.
• Worker harness - agents_md, worker lifecycle hooks, skills, worker launch, trace/log capture, and METRIC name=value emission. This is the measured system you optimize.
• Autoharness CLI - owns init/run/log, timeout, checks, metric parsing, keep -> commit, loss -> revert, confidence, .auto/ structured state, loop hooks, and finalize. Do not hand-edit .auto/log.jsonl.
• OpenSpec SDD - owns proposal/spec/design/tasks for a candidate change. Drive it with non-interactive status --json, instructions --json, and validate --strict; archive only after the following metric gate keeps it.

The default optimization surface is:

• agents_md - worker-facing markdown instructions, role boundaries, routing, workflow rules, task heuristics, and coordination guidance.
• hooks - worker lifecycle automation that injects context, blocks unsafe exits, records state, prepares inputs, or enforces measurement discipline.
• skills portfolio - multiple focused skills, trigger criteria, routing metadata, procedures, examples, and supporting files that affect worker skill selection and execution.

Do not optimize system_prompt or append_system_prompt unless the session explicitly puts them in scope.

When the hypothesis is skill-related, optimize a portfolio rather than one catch-all skill. Use a router skill or agents_md rule only to direct the worker into focused skills. Split unrelated task families into separate skills so later iterations can keep, discard, or ablate each behavior independently.

The autoharness CLI is the contract layer. It does not invent hypotheses or run the worker directly outside the measurement contract. It only invokes benchmarks, enforces timeout and checks gates, records structured state, commits real wins, reverts losses, computes confidence, and keeps the optim loop alive through loop hooks.

The optimization loop is ordered: metric keep/discard gate -> metric and worker trace/log audit -> OpenSpec SDD -> new run. The primary metric tells you whether an iteration won; the worker trace tells you why; OpenSpec SDD structures the next candidate harness change.

Contract Commands

Call these via Bash. Add --json when you need to parse the result.

• autoharness init --name N --metric M [--unit U] [--direction lower|higher] Once per task target. Writes the config header. Re-run only when the target or metric changes; that opens a new segment and resets the baseline.
• autoharness run [--timeout S] [--checks-timeout S] Runs the measurement contract (just measure, else .auto/measure.sh), which launches the worker and emits METRIC name=value lines. The CLI invokes it under timeout, parses metrics, saves raw output, writes .auto/latest-run-notice.md, and runs checks (just check, else .auto/checks.sh). Check time is excluded from the primary metric. This is the only source of truth for numbers. A new run is refused until the previous notice's evidence_audit file exists.
• autoharness log --metric N --status keep|discard|crash|checks_failed --desc D [--metrics JSON] [--asi JSON] Records the run. keep auto-commits; everything else auto-reverts code changes while preserving .auto/. You cannot keep if checks failed.
• finalize/finalize.sh <groups.json> Turns kept commits into clean review branches when the optimization is done.

Do not run git bookkeeping by hand during the loop. autoharness log owns commit and revert. Do not hand-edit .auto/log.jsonl.

OpenSpec SDD Driver

Use OpenSpec directly for each iteration's harness change. Do not invent a parallel .auto/changes protocol.

Required OpenSpec discipline:

• Bootstrap once per project if needed: openspec init --tools <tool-id>. Never run bare interactive openspec init in the loop.
• Create/select a change for the next harness hypothesis: openspec new change <kebab-name> or infer the one active change.
• Treat openspec status --change <name> --json as the state machine.
• Treat openspec instructions <artifact|apply> --change <name> --json as the artifact contract. Read its dependencies/context files, write only the requested artifact, and do not copy instruction/context blocks into outputs.
• Use openspec validate <name> --strict as the SDD correction oracle. Fix precise validation errors and retry.
• During apply, complete OpenSpec tasks and flip task checklist items as work is done.
• Do not archive before measurement. openspec archive <name> -y is allowed only after the following metric gate keeps the change. Do not archive discarded, crashed, or checks-failed metric losses merely because the OpenSpec artifacts validate.

OpenSpec SDD owns shape and validity of the candidate change. Autoharness owns metric survival. Archive only after both OpenSpec validation and metric keep.

Phase 1 - Frame The Harness Task

Do this once before looping.

1. Identify the HCP task, target metric, direction, timeout, and pass/fail checks. Specify how the measurement contract launches the worker. If the user says a run over 30 minutes is a timeout, encode that in the measurement contract.
2. Identify the exact harness files in scope. Default to agents_md, worker hooks, and skills portfolio files. Record off-limits files explicitly.
3. Read the current harness and task runner. Understand how the metric is produced before changing anything.
4. Locate the worker trace artifacts produced by the task runner. Record where traces live, how to read them, and which fields show model actions, tool calls, hook output, skill selection, errors, and final judgment.
5. Create a branch named autoharness/<goal>-<date>.
6. Write .auto/prompt.md. A fresh optim agent must be able to resume from it. Include objective, metric, command, timeout, checks, in-scope files, off-limits files, hard constraints, trace locations, current best, and a living Trace Findings / What's Been Tried section.
7. Write the measurement contract: just measure or .auto/measure.sh. It must emit the primary METRIC name=value line. For noisy tasks, report a median or pass-rate over repeated trials.
8. Add just check or .auto/checks.sh when syntax, schema, packaging, or harness invariants matter.
9. Run autoharness init ..., then autoharness run, then autoharness log ... for the baseline.

Metric And Trace/Log Audit

After every measured run, close the metric gate first, then inspect the metric record, .auto/latest-run-notice.md, worker trace, and benchmark log files before picking the next OpenSpec SDD change. Extract evidence, not impressions.

The notice names the raw run output and the evidence_audit file you must write. That file is the handoff from measurement to optimization: include the metric, job/artifact paths, trace/log paths read, concrete verifier failures or worker behaviors observed, and the hypothesis they support. Do not start the next autoharness run before this file exists.

Classify the main failure or win driver:

• framing - the worker misunderstood the task, success criteria, constraints, or priority order; usually points to agents_md.
• routing - the worker failed to select the right skill, selected it too late, or selected it for the wrong case; usually points to skill metadata, descriptions, or agents_md routing rules.
• procedure - the right skill was selected but the steps, examples, or decision rules were insufficient; usually points to skill content.
• context - required state, files, logs, benchmark details, or previous findings were absent or poorly surfaced; usually points to worker hooks or .auto/prompt.md structure.
• tooling - the trace shows wrong commands, missing checks, timeout handling mistakes, parse errors, or brittle measurement behavior; usually points to worker hooks, checks, or measurement scripts.
• stopping - the worker ended early, failed to recover, or looped unproductively; usually points to worker hooks and agents_md loop rules.

When logging, put trace-derived learning in --asi: cite the trace artifact, the observed behavior, the suspected root cause, and the next hypothesis. If a change is kept, the trace should explain why the metric moved. If discarded, the trace should explain what was ruled out.

Phase 2 - Gate, Audit, SDD, Run

Each iteration should test one coherent behavioral hypothesis. Prefer scoped OpenSpec SDD changes with clear causal stories over broad rewrites.

Use this decision order:

1. Close the previous run with autoharness log. Use the primary metric and checks result to choose keep, discard, crash, or checks_failed.
2. Audit .auto/log.jsonl, the latest metric output, worker trace artifacts, benchmark log files, failures, and asi.
3. Choose the most promising bottleneck in the HCP harness:
   • the worker receives the wrong task framing -> try agents_md;
   • the worker misses or misuses an available procedure -> try the skills portfolio by editing focused skills and routing metadata;
   • the right context/state is not available at the right worker lifecycle point -> try hooks;
   • the behavior crosses boundaries -> change the smallest coherent set across agents_md, hooks, and skills.
4. Drive the next harness change through OpenSpec SDD: openspec status --change <name> --json, openspec instructions <artifact|apply> --change <name> --json, artifact writes, implementation tasks, and openspec validate <name> --strict.
5. Tie the OpenSpec change to audit evidence. Name the trace/log failure mode and the behavior the change is expected to alter.
6. Run autoharness run only after the OpenSpec SDD change validates.
7. Do not decide survival yet. The next iteration begins by gating this new run with autoharness log. If and only if it is kept, archive the OpenSpec change with openspec archive <name> -y.
8. Periodically update .auto/prompt.md with durable trace/log findings and append deferred ideas to .auto/ideas.md.

Confidence matters. The reported confidence is best improvement divided by the noise floor. Treat values below 1x as noise and values around or above 2x as more likely real; rerun suspicious wins before building on them.

The Stop hook will push the next iteration automatically. Keep going until the goal is met, gains have bottomed out, or credible ideas are exhausted.

Phase 3 - Judge Done And Deliver

When the optimization is ready:

1. Group kept commits into reviewable units. Preserve application order. Merge groups that touch the same file or have tight dependencies.
2. Write groups.json using finalize/FINALIZE.md.
3. Run bash finalize/finalize.sh <groups.json>.
4. Create .auto/.done so the hook stops the loop.
5. Report branches, metric improvement, checks run, and any remaining ideas.

Finalize only packages the kept changes. It does not decide whether they are worth shipping; that judgment is yours.