test-driven-solve

name	test-driven-solve
description	Use the task's own verifier as your tight feedback loop. Make the smallest plausible change, run the verifier, read its output, iterate. Do not modify tests to make them pass.

Test-Driven Solve

The task has a verifier. That verifier is the graded contract. Your workflow should orbit it.

The loop

1. Run the verifier on the untouched repo. Capture output. This is your baseline.
2. Locate the narrowest file or function that, if changed correctly, would flip a verifier assertion from fail to pass.
3. Make the smallest change you can articulate.
4. Re-run the verifier. Read its output top to bottom.
5. If it passed, stop. If it didn't, the new output tells you what still fails — loop back to step 2 on that narrower problem.

Do not bundle changes. Each edit should be causally tied to one verifier signal. Bundled edits make regressions impossible to bisect.

Read the verifier's code

If the verifier is a script (test.sh, test_state.py, a pytest file), read it. It will tell you exactly what files it reads, what output strings it greps for, and which exit codes count as pass. This is often faster than staring at the brief.

Patterns to notice:

• grep -q "expected_string" output.txt → the literal substring matters; don't paraphrase.
• assert result == expected_list → order and element type matter.
• subprocess.run(["./my-binary", ...]) → the verifier calls your binary with specific args; your binary must accept them.

The rule that catches most cheats

Never modify the verifier or its fixtures to make it pass. You are not allowed to:

• Edit files under tests/, verifier/, tests_state/, or anything the brief labels as the grading harness.
• Change expected output constants.
• Hardcode the expected answer into a file the verifier reads (unless the task is literally asking you to produce that file as output — read the brief to tell the difference).

If a test looks wrong, re-read the brief. The test is almost never wrong; your model of the task is.

Partial credit and early stops

Some tasks grade on multiple criteria. When the verifier fails, look at which assertions failed — a failure on assertion 7 of 10 means assertions 1–6 are already passing, and you should be careful not to break them. Treat the passing set as constraints.

If the verifier produces a clear "you got this subset right" report, focus edits on the remaining subset only.

When you think you are done

• Re-run the verifier one final time from a clean shell state (cd back to /app, clear any env vars you set manually).
• Check you did not leave stray files (e.g. scratch.py, .bak copies) that the verifier might accidentally pick up.
• Only then call the harness's completion / finish tool.