mutation-testing

name	mutation-testing
description	Use before trusting any green test — prove the test can actually FAIL by mutating the code it covers. A test that still passes when its target is broken tested nothing.
origin	SkyyRose

Mutation Testing

A passing test is worthless if it cannot fail. Green is not evidence — it is a claim. The only way to cash that claim is to break the code the test claims to cover and watch the test catch it. If the test stays green while the code is provably broken, the test never exercised the code; it was decoration, not verification. This session proved two real fixes this way: an AES fail-closed guard and a brand-assets bulk-ingestion job.

Boot first: read the canonical sources — SOT.md → .wolf/anatomy.md → .wolf/cerebrum.md → .wolf/buglog.json → CLAUDE.md. Do not trust a green suite from memory; re-run it this session before mutating anything.

When to Use

• Before marking any bug fix or new guard "done" — pair with [[fail-closed-audit]] when the code under test is a gate/validator.
• Any test whose assertion you did not personally watch fail at least once.
• A test that was written to match the implementation instead of the requirement (the classic trap: code and test were authored together and agree by construction).
• Reviewing someone else's "tests pass" claim before accepting a PR or a subagent's report.

Method

1. Pick a critical assertion — the one line that, if false, means the fix does not hold. Not every assertion in the file; the one that carries the claim.
2. Apply the smallest plausible mutation to the code it covers — the kind of bug a careless edit would actually introduce. Real examples from this repo:
   • AES fail-closed: elif os.getenv("ENVIRONMENT", "").lower() == "production": raise EncryptionError(...) mutated to elif False: — deletes the guard without touching surrounding structure.
   • Brand-assets bulk ingestion finalize branch: if job_row.failed == 0: mutated to if job_row.failed >= 0: (always true), and the counter job_row.failed += 1 mutated to job_row.failed += 0 (never increments) — two independent ways the same branch could silently stop tracking failures.
3. Run only that test — not the whole suite. Isolate signal.
4. It MUST fail. If it passes, the test is inert — go to Guardrails, do not proceed.
5. Restore the mutation immediately and re-run to confirm baseline green. A mutation left in place is a bug you just introduced, not a finding.

Loop until every assertion is proven

Bounded, like [[drive-to-green]] — never more than 5 turns per assertion, stop if the same result repeats twice (that is guessing, not testing):

1. Pick the next critical assertion.  2. Mutate the smallest plausible thing it covers.
3. Run that one test → does it FAIL?  4. If it passes → the test is inert, fix the test.
5. Restore → confirm baseline green.  Repeat for every critical assertion. Stop when all
   are proven falsifiable.

Verify from an authoritative source

The proof is only real if the failure was observed, never assumed:

• Run the mutated code and read the output — do not predict the failure, capture it. The AES mutation produced the literal assertion message DID NOT RAISE EncryptionError; the brand-assets mutation produced assert 'completed' == 'partial'. Cite the exact failing message, not a paraphrase of what you expected it to say.
• A mutation that leaves the test green proves the test is inert — it proves nothing about whether the original code is right. Do not read "still passes" as "code is correct"; read it as "this test cannot see this bug."
• Restore-and-reconfirm is not optional — baseline green after restoring the mutation is the other half of the proof; skipping it means you don't know whether you fixed the mutation or broke something else.

Adversarial pass

• [[adversarial-verification]] — before calling an assertion proven, have a skeptic propose the mutation you would miss: an off-by-one in the opposite direction, a boundary condition (== vs >=), a counter that increments the wrong variable, an error swallowed instead of raised. The two brand-assets mutations above are exactly this pattern — same branch, two different ways to defeat it.
• If the skeptic's mutation also gets caught, the test is stronger than the one you picked alone. If it slips through, you found a real gap — fix the test, not the mutation.

Guardrails · Handoff · Log

• Never weaken a test to make it pass under mutation — that is hiding the gap, not closing it. Strengthen the assertion until the mutation is caught.
• A test that stays green under every mutation you tried has not "passed the mutation test" — it has failed it. Fix the test before trusting the fix.
• Gate/guard code whose fail-open branch you're mutating → [[fail-closed-audit]] first, mutation-testing second, to confirm the fix both fails closed AND is falsifiable.
• Cross-plugin fixes (backend vs imagery vs theme) hand off per CROSS-PLUGIN.md, then re-verify here before trusting the receiving plugin's green.
• Bounded loops that don't converge → [[drive-to-green]]; final claim of "done" still needs [[verification-loop]].
• Log every inert-test finding to .wolf/buglog.json (tag mutation-testing, cross-link bug-230 lineage if the inert test was covering a fail-open gate), and record the lesson via [[continuous-learning]].