harness-optimization-loop

# Harness Optimization Loop ## Overview Improve the system around the model, not the model weights: prompts, tool order, context selection, verification gates, routing, memory use, and workflow shape. Optimize only when there is an evaluation surface. ## When to Use Use this skill: - A prompt, workflow, or tool sequence repeatedly underperforms. - A task needs a more reliable agent harness before scaling or reuse. - There is a benchmark, fixture set, CI check, golden output, human rubric, or repeatable manual test. - The user asks to optimize an agent workflow, prompt, harness, evaluator, or multi-agent process. Do not use it when there is no way to compare candidates, when a single obvious bug fix is available, or when the change would create broad platform complexity for a narrow one-off task. ## Workflow 1. Define the target behavior. - What should improve? - What must not regress? - What evidence will decide between candidates? 2. Capture the baseline. - Save the current prompt, workflow, config, or instructions. - Run the existing evaluation or reproduce the failure. - Record the result and the exact command or manual check. 3. Isolate variables. - Change one dimension per candidate when possible. - Separate bug fixes from prompt wording changes. - Separate retrieval/context changes from tool-routing changes. 4. Generate small candidates. - Prefer 2-4 candidates. - Keep each candidate cheap to test and easy to revert. - Avoid adding new abstractions unless the evaluation demands them. 5. Evaluate and compare. - Use the same inputs and scoring method for every candidate. - Record pass/fail, score, cost, latency, and notable failure modes when available. - Prefer evidence over intuition. 6. Select, integrate, and verify. - Keep the best candidate or combine only independently validated changes. - Run the project's normal verification gate. - Archive enough notes for the next iteration. ## Candidate Log Template ```text Goal: Baseline: Evaluation: Candidate A: - Change: - Result: - Failure mode: Candidate B: - Change: - Result: - Failure mode: Decision: Follow-up: ``` ## What Counts As Harness - System or developer instructions. - Prompt templates and examples. - Retrieval, memory, and context selection. - Tool choice and call order. - Subagent routing and review loops. - Verification gates and scoring scripts. - File-system trace layout for future iterations. ## Common Mistakes - Optimizing without a metric or repeatable check. - Changing prompt, tool order, and evaluator at the same time. - Keeping a candidate because it feels cleaner while the score regresses. - Summarizing historical traces so aggressively that debugging evidence is lost. - Treating this as model training. This skill changes the harness only.