Use this when a benchmark delta hides churn: one config gains some cells and loses others. The unit is a paired cell: same task, same rep, same model/thinking unless the comparison explicitly changes them.
-
Lock the comparison. State left config, right config, subset, reps, model, thinking level, and result roots. Confirm both sides have the intended cells. Completion: every compared pair maps to exactly one left result.json and one right result.json.
-
Split net from churn. Compute left-only solves, right-only solves, both solved, neither solved, mean/median partial delta, token/cost/wall/tool deltas, and difficulty/language splits. Completion: the report shows both net solve delta and solve-flip counts.
-
Build a trajectory packet for each flip. For every solve flip, gather the paired result.json, session JSONL, model.patch, verifier artifacts, changed-file list, patch stats, tool timeline, and config-specific tool/extension traces. Completion: each flip has a Markdown or JSON packet that can be reviewed without re-running the benchmark.
-
Map verifier failure to patch delta. For each lost solve, identify whether failure is f2p-only, p2p regression, apply/timeout/empty-patch, or verifier artifact. Then connect the failing test or reward drop to the concrete patch difference. Completion: every classified loss names the failed test/invariant and the exact patch behavior that explains it; uncertainty is explicit.
-
Classify the driver. Use the smallest specific bucket that fits: wrong seam/layer, under-implementation, over-implementation, missing invariant/guard, protocol/interface drift, cross-scope regression, validation gap, resource exhaustion, or likely variance. Completion: every flip has one primary bucket, optional secondary bucket, and evidence bullets.
-
Compare winning and losing patterns. Do not infer skill guidance from losses alone. Run the same packet method on right-only wins, then compare recurring patterns in wins vs losses. Completion: the synthesis separates “keep” patterns from “prevent” patterns.
-
Translate to skill-design hypotheses. Use writing-great-skills principles: propose checkable process changes, not vague advice. Completion: each proposed guidance change has a trigger, an action, and a completion criterion that would have changed at least one observed trajectory.
-
Publish as an evidence-first report. For this repo, produce a self-contained HTML report served on the Tailnet. Include the packet links, bucket table, concrete task examples, and a short conclusion. Completion: the URL works and the report distinguishes direct session evidence, source/patch evidence, and inference.
