| name | test-driven-solve |
| description | Use the task's own verifier as your tight feedback loop. Make the smallest plausible change, run the verifier, read its output, iterate. Do not modify tests to make them pass. |
Test-Driven Solve
The task has a verifier. That verifier is the graded contract. Your workflow should
orbit it.
The loop
- Run the verifier on the untouched repo. Capture output. This is your baseline.
- Locate the narrowest file or function that, if changed correctly, would flip a
verifier assertion from fail to pass.
- Make the smallest change you can articulate.
- Re-run the verifier. Read its output top to bottom.
- If it passed, stop. If it didn't, the new output tells you what still fails —
loop back to step 2 on that narrower problem.
Do not bundle changes. Each edit should be causally tied to one verifier signal.
Bundled edits make regressions impossible to bisect.
Read the verifier's code
If the verifier is a script (test.sh, test_state.py, a pytest file), read it.
It will tell you exactly what files it reads, what output strings it greps for, and
which exit codes count as pass. This is often faster than staring at the brief.
Patterns to notice:
grep -q "expected_string" output.txt → the literal substring matters; don't
paraphrase.assert result == expected_list → order and element type matter.subprocess.run(["./my-binary", ...]) → the verifier calls your binary with
specific args; your binary must accept them.
The rule that catches most cheats
Never modify the verifier or its fixtures to make it pass. You are not allowed
to:
- Edit files under
tests/, verifier/, tests_state/, or anything the brief
labels as the grading harness.
- Change expected output constants.
- Hardcode the expected answer into a file the verifier reads (unless the task is
literally asking you to produce that file as output — read the brief to tell
the difference).
If a test looks wrong, re-read the brief. The test is almost never wrong; your
model of the task is.
Partial credit and early stops
Some tasks grade on multiple criteria. When the verifier fails, look at which
assertions failed — a failure on assertion 7 of 10 means assertions 1–6 are already
passing, and you should be careful not to break them. Treat the passing set as
constraints.
If the verifier produces a clear "you got this subset right" report, focus edits
on the remaining subset only.
When you think you are done
- Re-run the verifier one final time from a clean shell state (
cd back to
/app, clear any env vars you set manually).
- Check you did not leave stray files (e.g.
scratch.py, .bak copies) that the
verifier might accidentally pick up.
- Only then call the harness's completion / finish tool.
