mutation-testing

# Mutation Testing For writing good tests (factories, behavior-driven patterns), load the `testing` skill. This skill focuses on verifying test effectiveness. Mutation testing answers the question: **"Are my tests actually catching bugs?"** Code coverage tells you what code your tests execute. Mutation testing tells you if your tests would **detect changes** to that code. A test suite with 100% coverage can still miss 40% of potential bugs. --- ## Core Concept **The Mutation Testing Process:** 1. **Generate mutants**: Introduce small bugs (mutations) into production code 2. **Run tests**: Execute your test suite against each mutant 3. **Evaluate results**: If tests fail, the mutant is "killed" (good). If tests pass, the mutant "survived" (bad - your tests missed the bug) **The Insight**: A surviving mutant represents a bug your tests wouldn't catch. --- ## When to Use This Skill Use mutation testing analysis when: - Reviewing code changes on a branch - Verifying test effectiveness after TDD - Identifying weak tests that appear to have coverage - Finding missing edge case tests - Validating that refactoring didn't weaken test suite **Integration with TDD:** ``` RED-GREEN-MUTATE-REFACTOR Cycle ┌─────────────────────────────────────────────────┐ │ 1. RED: Write failing test │ │ 2. GREEN: Minimum code to pass │ │ 3. MUTATE: Verify tests catch real bugs ◄── │ ← You are here │ 4. REFACTOR: Improve structure with confidence │ └─────────────────────────────────────────────────┘ ``` **Why MUTATE before REFACTOR:** Mutation testing validates test strength *before* you restructure code. Refactoring with unverified tests means restructuring code whose safety net you haven't checked. --- ## Execution Process When verifying test effectiveness, **actually mutate the code and run the tests.** Do not just reason about whether tests would catch mutations — prove it. ### Step 1: Identify Changed Code ```bash # Get files changed on the branch git diff main...HEAD --name-only | grep -E '\.(ts|js|tsx|jsx)$' | grep -v '\.test\.' # Get detailed diff for analysis git diff main...HEAD -- src/ ``` ### Step 2: Apply Mutations and Run Tests For each changed function/method, work through the mutation operators (see Mutation Operators section below). For each applicable mutation: 1. **Mutate**: Change the production code (e.g., flip `*` to `/`, negate a condition) 2. **Run**: Execute the test suite 3. **Evaluate**: Did a test fail? - **Yes** → mutant killed (good). Revert the mutation. - **No** → mutant survived (bad). Revert the mutation, then add or strengthen a test. 4. **Revert**: Always restore the original code before the next mutation **Always revert each mutation before applying the next.** Never leave mutated code in place. You do not need to apply every possible mutation to every line. Focus on: - Changed code on the branch - Operators most likely to have surviving mutants (see Quick Reference) - Conditions with boundary values - Boolean logic with multiple operands ### Step 3: Produce a Report After working through the mutations, produce a summary: ```markdown ## Mutation Testing Report ### Killed (tests caught the mutation) - `calculateTotal`: `*` → `/` — killed by "calculates total for multiple items" - `isEligible`: `>=` → `>` — killed by "returns true at exact boundary" ### Survived (tests DID NOT catch the mutation) - `applyDiscount`: `>` → `>=` — no test for boundary value at exactly 100 → **Action**: Add boundary test for discount threshold ### Summary - Mutations applied: 8 - Killed: 6 - Survived: 2 - Mutation score: 75% ``` ### Step 4: Kill Surviving Mutants Not every surviving mutant warrants a new test. Some mutations produce equivalent behavior, and some boundary cases are low-risk enough that the test would add noise without meaningful protection. **Fix immediately** when: - The mutation represents a realistic bug (wrong operator, inverted condition) - The surviving mutant is in critical business logic (money, permissions, eligibility) - The fix is a simple boundary test or stronger assertion **Ask the human** when: - You're unsure whether the mutation represents a real risk - The test to kill it would be complex or hard to name clearly - The mutation is in a code path that's also covered by integration/E2E tests - The surviving mutant feels like an equivalent mutant but you're not certain Present the mutation, explain why the current tests don't catch it, and let the human decide whether it's worth a new test. When fixing, follow TDD — write the failing test first, verify it fails against the mutated code, then verify it passes against the original code. --- ## Mutation Operators ### Arithmetic Operator Mutations | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `a + b` | `a - b` | Addition behavior | | `a - b` | `a + b` | Subtraction behavior | | `a * b` | `a / b` | Multiplication behavior | | `a / b` | `a * b` | Division behavior | | `a % b` | `a * b` | Modulo behavior | **Example Analysis:** ```typescript // Production code const calculateTotal = (price: number, quantity: number): number => { return price * quantity; }; // Mutant: price / quantity // Question: Would tests fail if * became /? // ❌ WEAK TEST - Would NOT catch mutant it('calculates total', () => { expect(calculateTotal(10, 1)).toBe(10); // 10 * 1 = 10, 10 / 1 = 10 (SAME!) }); // ✅ STRONG TEST - Would catch mutant it('calculates total', () => { expect(calculateTotal(10, 3)).toBe(30); // 10 * 3 = 30, 10 / 3 = 3.33 (DIFFERENT!) }); ``` ### Conditional Expression Mutations | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `a < b` | `a <= b` | Boundary value at equality | | `a < b` | `a >= b` | Both sides of condition | | `a <= b` | `a < b` | Boundary value at equality | | `a <= b` | `a > b` | Both sides of condition | | `a > b` | `a >= b` | Boundary value at equality | | `a > b` | `a <= b` | Both sides of condition | | `a >= b` | `a > b` | Boundary value at equality | | `a >= b` | `a < b` | Both sides of condition | **Example Analysis:** ```typescript // Production code const isAdult = (age: number): boolean => { return age >= 18; }; // Mutant: age > 18 // Question: Would tests fail if >= became >? // ❌ WEAK TEST - Would NOT catch boundary mutant it('returns true for adults', () => { expect(isAdult(25)).toBe(true); // 25 >= 18 = true, 25 > 18 = true (SAME!) }); // ✅ STRONG TEST - Would catch boundary mutant it('returns true for exactly 18', () => { expect(isAdult(18)).toBe(true); // 18 >= 18 = true, 18 > 18 = false (DIFFERENT!) }); ``` ### Equality Operator Mutations | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `a === b` | `a !== b` | Both equal and not equal cases | | `a !== b` | `a === b` | Both equal and not equal cases | | `a == b` | `a != b` | Both equal and not equal cases | | `a != b` | `a == b` | Both equal and not equal cases | ### Logical Operator Mutations | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `a && b` | `a \|\| b` | Case where one is true, other is false | | `a \|\| b` | `a && b` | Case where one is true, other is false | | `a ?? b` | `a && b` | Nullish coalescing behavior | **Example Analysis:** ```typescript // Production code const canAccess = (isAdmin: boolean, isOwner: boolean): boolean => { return isAdmin || isOwner; }; // Mutant: isAdmin && isOwner // Question: Would tests fail if || became &&? // ❌ WEAK TEST - Would NOT catch mutant it('returns true when both conditions met', () => { expect(canAccess(true, true)).toBe(true); // true || true = true && true (SAME!) }); // ✅ STRONG TEST - Would catch mutant it('returns true when only admin', () => { expect(canAccess(true, false)).toBe(true); // true || false = true, true && false = false (DIFFERENT!) }); ``` ### Boolean Literal Mutations | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `true` | `false` | Both true and false outcomes | | `false` | `true` | Both true and false outcomes | | `!(a)` | `a` | Negation is necessary | ### Block Statement Mutations | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `{ code }` | `{ }` | Side effects of the block | **Example Analysis:** ```typescript // Production code const processOrder = (order: Order): void => { validateOrder(order); saveOrder(order); sendConfirmation(order); }; // Mutant: Empty function body // Question: Would tests fail if all statements removed? // ❌ WEAK TEST - Would NOT catch mutant it('processes order without error', () => { expect(() => processOrder(order)).not.toThrow(); // Empty function also doesn't throw! }); // ✅ STRONG TEST - Would catch mutant it('saves order to database', () => { processOrder(order); expect(mockDatabase.save).toHaveBeenCalledWith(order); }); ``` ### String Literal Mutations | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `"text"` | `""` | Non-empty string behavior | | `""` | `"Stryker was here!"` | Empty string behavior | ### Array Declaration Mutations | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `[1, 2, 3]` | `[]` | Non-empty array behavior | | `new Array(1, 2)` | `new Array()` | Array contents matter | ### Unary Operator Mutations | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `+a` | `-a` | Sign matters | | `-a` | `+a` | Sign matters | | `++a` | `--a` | Increment vs decrement | | `a++` | `a--` | Increment vs decrement | ### Method Expression Mutations (TypeScript/JavaScript) | Original | Mutated | Test Should Verify | |----------|---------|-------------------| | `startsWith()` | `endsWith()` | Correct string position | | `endsWith()` | `startsWith()` | Correct string position | | `toUpperCase()` | `toLowerCase()` | Case transformation |