testsmith

# testsmith Generate tests that catch real defects, not tests that pad coverage. Built on the adversarial pattern: a test generator and a mutant generator push against each other until the suite survives the mutants. ## Why this exists (evidence) - AdverTest (test-gen agent vs mutant-gen agent, guided by coverage + mutation score) hit 66.63% fault detection on Defects4J: +8.56% over the best prior LLM method and +63.3% over EvoSuite, at comparable coverage. Better probing, not more tests. Source: arXiv:2602.08146. - The problem it fixes: 66% of devs say AI code is "almost right, but not quite"; 45% find debugging AI code more time-consuming; AI bug-introduction runs 9.8%-42.1%. And agents tend to emit an explosion of low-value LLM tests. Quantity is not the goal; fault detection is. ## The loop ### Step 0. Reproduce first (non-negotiable for bug fixes) Write ONE failing test that captures the bug or the missing spec, and run it to confirm it FAILS for the right reason, BEFORE any fix. A fix without a prior failing test is unverified. ### Step 1. Generate candidate tests (test-gen) Write the smallest set of tests that pin the intended behavior: the happy path, the boundary, and the specific failure mode. Assert on real outputs and identifiers, not on incidental formatting. ### Step 2. Mutate (mutant-gen, the adversary) Introduce small, plausible faults into the code under test (flip a comparison, off-by-one, drop a branch, swap operands, return early, null a value). Each mutant is a hypothesis: "a real bug could look like this." ### Step 3. Kill or learn Run the tests against each mutant: - Mutant killed (a test fails) -> good, that behavior is protected. - Mutant SURVIVES (all tests pass) -> a gap. Add ONE targeted test that kills it. Do not add unrelated tests. ### Step 4. Stop Stop when surviving mutants are only equivalent mutants (no behavior change) or the mutation score crosses your threshold. The output is a SMALL suite where every test earns its place by killing a mutant or pinning a spec. ## How to run it - Small scope: do the loop inline, a few mutants by hand. - Larger scope: orchestrate with a Workflow, a test-gen stage and a mutant-gen stage in a pipeline, looping until no fresh survivors (mirrors the find -> adversarially-verify pattern). Route the mechanical mutant generation to a cheaper model (see model-router); keep assertion design on the stronger model. - If the project has a mutation-testing tool (Stryker, mutmut, PIT, cosmic-ray), use it to score instead of hand-mutating. ## Output contract - A failing test existed before the fix (for bug work). - Final suite is SMALL and every test maps to a killed mutant or a named spec. - Report mutation result: mutants tried, killed, survived (and why survivors are equivalent). Do not report a coverage percentage as if it were fault detection. ## Composes with - `test-driven-development` / `adversarial-verify`: testsmith is the test-quality engine under them. - `model-router`: mutant generation is mechanical -> cheap model; assertion design -> stronger model. ## Honest limits - Mutation testing approximates real faults; it does not prove correctness. Equivalent mutants need human judgment. - The AdverTest numbers are on Java/Defects4J; the pattern transfers but your delta depends on language and harness. Measure mutants-killed, not lines-covered.