test-the-tests

name	test-the-tests
description	Verify a test actually falsifies the behavior it claims to cover, via real mutation. Use right after writing or changing a test, when auditing an existing suite's real coverage, or as a step inside a code review.

Test the Tests

Passing and high coverage are not evidence, whoever wrote the test — including you, just now. A test proves it runs; it does not prove it discriminates correct from incorrect behavior. Ask:

What fault would this test fail to catch?

The mutation loop

For each test that carries real weight (guards a fix, a regression, an invariant — not a trivial getter):

1. Pick the smallest plausible fault: invert a condition, change < to <=, remove a validation, suppress an error, skip a side effect, hardcode a return value, revert to the prior (buggy) behavior.
2. Apply it to the implementation, run the test, confirm it fails for the right reason — not an unrelated crash.
3. Revert immediately, confirm the tree is clean (git status/git diff), confirm the suite passes again. NEVER leave mutated code in the repository between steps.

Copy the file before mutating it (cp f f.bak) and restore from the copy. NEVER restore with git checkout/git restore while the work under test is still uncommitted — those discard the work along with the mutation. Commit first, or use the copy.

If running the mutation isn't practical, name the smallest mutation that should be tried instead of skipping the exercise — that name is itself the finding.

"Not practical" is a conclusion, not an assumption — establish it as AGENTS.md requires (is the runtime actually missing, is a Linux host available) before falling back to naming the mutation. A "can't run it here" that turns out to be wrong ships tests nobody has ever seen fail.

Common ways a test looks strong but isn't

• A real daemon, cluster or network is one mutation away. A test that reaches the code under test through a validation error — "the driver is rejected, so docker buildx create is never called" — has its only barrier inside the thing being mutated. Invert that validation and the test provisions real infrastructure, leaves it behind, and hangs instead of failing when nothing drains what the code writes. Neutralize the escape before relying on it (an empty PATH, an unroutable address) and drain every pipe the code writes to, so the fault fails the test rather than the machine running it.
• The assertion holds under the bug too. A chain assertion like index(a) < index(b) < index(c) can pass under both the fix and the regression it's meant to catch if the scenario doesn't force them to disagree. Reshape the scenario (add a case whose correct answer differs from what the bug would produce) instead of trusting a single run.
• It re-asserts a second recording of the same event instead of a property true by construction — comparing two independently-recorded orderings is racier and weaker than asserting a monotonic counter or a guaranteed dependency order.
• It exercises the unit in isolation, never the real wiring a regression would actually break. Prefer driving the real entry point end-to-end when the risk is in the wiring.
• It asserts implementation trivia (mock called N times, internal helper ran) instead of externally observable behavior.
• Coverage number, not falsifiability. A line being executed says nothing about whether a wrong value on that line would be caught.
• An extended test quietly drops what the old one proved. Adding cases to a fixture can remove the conflict that made an earlier property observable. After editing an existing test, re-run the mutations the previous version caught, not only the new ones.
• Ordered behavior tested without conflicts. For precedence lists and fallback chains, a fixture with one candidate per level survives swapping adjacent candidates. Give every level two candidates whose effects differ.

Output

For each test verified this way: the fault tried, whether it failed as expected, and if not, the smallest change that would make it discriminate. For tests not mutated, name the smallest fault that should be tried next — don't assert confidence without it.