harness-optimization-loop

name	harness-optimization-loop
description	Use when improving an agent harness, prompt, workflow, tool order, retrieval strategy, subagent routing, or evaluation loop where multiple candidate approaches can be compared against observable results.

Harness Optimization Loop

Overview

Improve the system around the model, not the model weights: prompts, tool order, context selection, verification gates, routing, memory use, and workflow shape. Optimize only when there is an evaluation surface.

When to Use

Use this skill:

• A prompt, workflow, or tool sequence repeatedly underperforms.
• A task needs a more reliable agent harness before scaling or reuse.
• There is a benchmark, fixture set, CI check, golden output, human rubric, or repeatable manual test.
• The user asks to optimize an agent workflow, prompt, harness, evaluator, or multi-agent process.

Do not use it when there is no way to compare candidates, when a single obvious bug fix is available, or when the change would create broad platform complexity for a narrow one-off task.

Workflow

1. Define the target behavior.

   • What should improve?
   • What must not regress?
   • What evidence will decide between candidates?

2. Capture the baseline.

   • Save the current prompt, workflow, config, or instructions.
   • Run the existing evaluation or reproduce the failure.
   • Record the result and the exact command or manual check.

3. Isolate variables.

   • Change one dimension per candidate when possible.
   • Separate bug fixes from prompt wording changes.
   • Separate retrieval/context changes from tool-routing changes.

4. Generate small candidates.

   • Prefer 2-4 candidates.
   • Keep each candidate cheap to test and easy to revert.
   • Avoid adding new abstractions unless the evaluation demands them.

5. Evaluate and compare.

   • Use the same inputs and scoring method for every candidate.
   • Record pass/fail, score, cost, latency, and notable failure modes when available.
   • Prefer evidence over intuition.

6. Select, integrate, and verify.

   • Keep the best candidate or combine only independently validated changes.
   • Run the project's normal verification gate.
   • Archive enough notes for the next iteration.

Candidate Log Template

Goal:
Baseline:
Evaluation:

Candidate A:
- Change:
- Result:
- Failure mode:

Candidate B:
- Change:
- Result:
- Failure mode:

Decision:
Follow-up:

What Counts As Harness

• System or developer instructions.
• Prompt templates and examples.
• Retrieval, memory, and context selection.
• Tool choice and call order.
• Subagent routing and review loops.
• Verification gates and scoring scripts.
• File-system trace layout for future iterations.

Common Mistakes

• Optimizing without a metric or repeatable check.
• Changing prompt, tool order, and evaluator at the same time.
• Keeping a candidate because it feels cleaner while the score regresses.
• Summarizing historical traces so aggressively that debugging evidence is lost.
• Treating this as model training. This skill changes the harness only.