improve-agent-harness

name	improve-agent-harness
description	Improve an agent harness through bounded experiments over a batch of traces or evaluations: cluster failures, classify layers, test one harness-variable hypothesis, cold-replay candidates, and require held-out or negative regression evidence. Use when recurring agent failures warrant controlled harness improvement rather than diagnosing one surprising run.
license	MIT

Improve an Agent Harness

Run a bounded experiment sequence to improve a harness from a representative batch of traces or evaluations. A harness is the agent-operating surface around the model: instructions, context selection, tools and contracts, orchestration, graders, memory, and retrieved skills. This skill improves that surface; it does not treat one surprising run as a population-level defect.

Establish the experiment contract

Before changing a harness variable, record:

Target outcome and baseline batch:
Trace/evaluation provenance, privacy, and allowed use:
Failure clustering method and inclusion rule:
Layer classification:
Primary metric and acceptance threshold:
Guard metrics and prohibited regressions:
Editable harness surfaces and sole writer:
Parent and child Route Records, when applicable:
Experiment / time / cost budget:
Cold-replay environment and held-out / negative set:
Checkpoint and rollback boundary:
Human release reviewer:
Final evidence:

Use a batch with enough representative examples to distinguish recurring failure from a one-off. Preserve the candidate's identity, input version, tool availability, and evaluation configuration. If external evaluation data, prompts, tool contracts, deployment, or release authority is missing, limit work to authorized read-only analysis and stop before that boundary.

Cluster before selecting a variable

For every included failure, record the trace/evaluation identity, observable failure signature, affected task class, evidence, and confidence. Cluster only by a shared observable signature or a tested causal hypothesis. Classify each cluster's leading layer:

• instructions;
• context selection;
• tools or contracts;
• model routing and context forks;
• orchestration;
• graders;
• memory; or
• retrieved skills.

Keep alternate layers visible when the evidence does not discriminate them. Choose the highest-impact cluster that fits the budget and can be evaluated without broadening authority.

Test one coherent hypothesis at a time

State each experiment as:

Changing <one harness variable> should reduce <failure cluster> on <batch>
without worsening <guard> because <causal hypothesis>.

Apply the smallest candidate change within the authorized harness surface. Replay it cold: no hidden carry-over from an interactive session, previous memory state, cached result, altered data, or manually repaired tool result. Evaluate the same baseline subset and an independent held-out or negative set. The held-out or negative set must challenge the likely regression, such as a nearby task class, safety boundary, tool-call shape, or previously successful behavior.

When the candidate layer is instructions or retrieved skills, consult writing-for-agents for the craft of the edit: pointer wording, completion criteria, leading words, and pruning decide whether the smallest candidate is a wording change, a disclosure move, or a deletion. It supplies craft, not process; clustering, attribution, cold replay, and keep/discard evidence stay with this skill.

Test route hypotheses

When the candidate changes a model route, invoke route-agent-models for each candidate. Preserve the returned Route Records with the experiment evidence.

Change one route variable at a time: provider/model, reasoning effort, service tier, role, context fork, tool surface, or fallback. Keep task briefs, permissions, fixtures, acceptance criteria, and child-spawn policy constant.

Measure quality separately from latency and cost. Require guard metrics for tool correctness, authority, route mismatch, cancellation, and partial failure. Treat silent substitution or unknown effective configuration as a discard.

Keep or discard with evidence

Keep a candidate only when the primary threshold improves and all declared guards hold on cold replay and held-out/negative evidence. Otherwise discard only the experiment-owned change and record why. One candidate must not bundle prompt edits, tool changes, evaluator changes, and orchestration changes unless they form one inseparable harness variable; if inseparable, document why the experiment remains attributable.

Maintain an experiment ledger:

experiment | cluster | layer | hypothesis | candidate | baseline replay
cold replay | requested/effective route | held-out/negative result
guard result | keep/discard | evidence

Stop on the target, budget exhaustion, repeated unchanged evidence, a plateau, invalid or unavailable evaluation, new authority, overlapping user edits, or user cancellation. A score improvement with weaker guards or an altered grader is a discard, not progress.

Review before release

After the last kept candidate, produce a release-review packet with the baseline, clusters considered, changes kept and discarded, exact replay and held-out evidence, residual risk, rollback boundary, and required deployment authority. A human reviewer must approve release separately. Do not deploy, publish, modify external prompts or tools, or promote external evaluation data under this skill unless that separate authority is explicitly granted.

Boundary with adjacent skills

evaluate-agent-workflow diagnoses one agent workflow end to end and explains where its evidence points; use it for a single surprising or unclear behavior. hill-climbing optimizes one mechanical metric generally. This skill owns batch-derived failure clusters, harness-layer classification, cold replay, and held-out/negative regression evidence. It does not invoke those companions or reuse their procedures as hidden execution.

Portability

Keep the contract technology- and client-neutral. Describe models, tools, evaluators, traces, and memory by their observable interfaces and authority; do not assume a named framework, provider, or agent client.

Examples

Improve the harness for recurring invalid tool arguments across 40 recorded
evaluations. You may edit only tool-schema descriptions and instruction text;
use 10 held-out evaluations and stop after four experiments.

Compare an inherited worker route with one explicit economical route. Hold the
task, context fork, tools, and acceptance rubric constant.

Analyze a batch of long-context failures and propose bounded experiments, but
do not alter prompts, retrieved skills, or deployments.

Counterexamples

• “This one answer was odd; tune the prompt” is a single-run diagnosis; first reproduce or evaluate it, then collect a representative batch if it recurs.
• “Raise the benchmark score by changing the grader and prompt together” loses attribution and weakens the oracle; preserve the evaluator and test one harness variable.