| name | harness-stripping |
| description | Systematically remove one harness component at a time and measure impact, killing scaffolding that no longer earns its complexity. |
| when_to_use | auditing a harness after a model upgrade to see what workarounds are now obsolete, a component encodes an assumption about model weakness worth re-testing, or the harness has grown organically and you suspect dead/redundant machinery |
Harness Stripping
Every harness component was added to compensate for a specific model failure. Models improve. Components don't retire themselves. The scaffolding that saved you on Sonnet 4.5 may be dead weight — or actively harmful — on Opus 4.6. Strip it deliberately, one piece at a time, and let evals tell you what still earns its keep.
Inspired by Prithvi's March 2026 harness post on evaluator-generator separation and the general "re-test your assumptions each model bump" discipline.
When to apply
- A model upgrade just landed and your harness was tuned for the previous generation.
- A component's justification is "we added this because the model used to do X" — and you haven't checked whether it still does X.
- The harness has accreted over months and nobody remembers what half the machinery is for.
- Cost or latency is climbing and you suspect redundant belt-and-suspenders layers.
Procedure
-
Inventory the components. List every distinct piece of scaffolding: prompt sections, tool wrappers, post-hoc validators, retry loops, evaluator personas, structured-output enforcers, sandbox rules. One row per component. Note the failure mode each was added to prevent.
-
Rank by suspicion. Put the components most likely to be obsolete at the top: anything added before the last two model bumps, anything targeting a failure mode you haven't seen recently, anything whose original justification is now folklore.
-
Pick a baseline eval. You need a repeatable metric before you touch anything. Reuse an existing eval set if you have one; otherwise pick 20–60 tasks representative of production work. Record baseline score, cost, and wall-clock.
-
Strip one component. Only one. Comment it out or gate it behind a flag — don't delete yet. Re-run the eval.
-
Compare against baseline.
- Score within noise, cost/latency down → the component is dead weight. Delete.
- Score drops measurably → the component still earns its complexity. Restore and note what failure mode returned.
- Score improves → the component was actively harmful. Delete and investigate why (often: over-constraining a now-capable model).
-
Commit the delta. Land the strip (or the restore-with-notes) as its own commit. Do not batch multiple strips into one change — you lose the ability to attribute the score movement.
-
Repeat for the next component. Re-establish baseline from the new state each round, not the original. Compounding strips have compounding effects.
Anti-patterns
- Stripping two components at once — you can't tell which one mattered. Halve the signal, double the confusion.
- Skipping the eval "because it's obviously safe to remove" — the harness accreted for reasons. Some are still real. Measure.
- Deleting instead of gating on the first pass — you will want to A/B mid-review. Flag first, delete after the eval confirms.
- Trusting anecdotes over the eval — "it feels better without it" is how load-bearing components get removed. If the eval doesn't show it, it isn't there.
- Stripping components that guard safety, sandboxing, or cost caps — those aren't compensating for model weakness. Leave them.
- Doing this on prod traffic — run against an eval set, not real users. The failure modes you're re-probing are exactly the ones that hurt users.
What to strip first
Highest yield in practice:
- Structured-output enforcers layered on top of models that now emit valid JSON natively.
- Multi-step "plan then execute" wrappers on tasks the model now one-shots.
- Retry loops around tool calls that no longer flake.
- Evaluator personas whose critiques the generator now anticipates on its own.
- Verbose "remember to do X" prompt sections where X is now default behavior.
When NOT to apply
Don't strip mid-project on a live long-running run — you'll perturb sessions in flight. Do it between projects, or on a forked branch. Also skip if you don't have an eval you trust; stripping without measurement is guessing.
Related
- [[shift-notes]] — record which components were stripped and when, so the next audit doesn't re-strip and re-restore the same piece.
- [[adversarial-verify]] — the evaluator-generator pattern that may itself be a strip candidate on newer models.
- [[broken-window-check]] — if you strip a component and the eval regresses in a specific way, that's your new broken window to hunt.
