testingmutation-testing

name	testing/mutation-testing
description	Mutation testing with Stryker to verify test quality by introducing code mutations and measuring detection rates
category	testing
tags	["testing","mutation","stryker","quality","coverage"]
related_skills	["testing/comprehensive-testing","testing/vitest","methodology/quality-gates"]

Mutation Testing

Measure test quality by introducing bugs (mutations) and verifying your tests catch them.

Quick Start

# Install Stryker
npm install -D @stryker-mutator/core @stryker-mutator/vitest-runner

# Initialize configuration
npx stryker init

# Run mutation testing
npm run test:mutation

Core Concept

Mutation testing introduces small changes (mutations) to your code and runs your tests. If tests still pass, they're too weak.

// Original code
function isAdult(age) {
  return age >= 18;
}

// Mutations Stryker creates:
function isAdult(age) { return age > 18; }   // >= to >
function isAdult(age) { return age <= 18; }  // >= to <=
function isAdult(age) { return false; }      // return false
function isAdult(age) { return true; }       // return true

A good test catches all mutations:

describe('isAdult', () => {
  it('returns true for age 18', () => {
    expect(isAdult(18)).toBe(true);  // Catches >= to >
  });

  it('returns false for age 17', () => {
    expect(isAdult(17)).toBe(false); // Catches >= to <=
  });

  it('returns true for age 100', () => {
    expect(isAdult(100)).toBe(true); // Catches return false
  });
});

Configuration

stryker.config.json

{
  "$schema": "https://raw.githubusercontent.com/stryker-mutator/stryker/master/packages/core/schema/stryker-schema.json",
  "packageManager": "npm",
  "testRunner": "vitest",
  "mutate": [
    "src/**/*.js",
    "src/**/*.ts",
    "!src/**/*.test.js",
    "!src/**/*.spec.ts"
  ],
  "reporters": [
    "progress",
    "clear-text",
    "html",
    "json"
  ],
  "htmlReporter": {
    "fileName": "reports/mutation/index.html"
  },
  "thresholds": {
    "high": 80,
    "low": 60
     
  
   
   

Mutation Operators

Arithmetic Operators

// Original: a + b
a - b    // Plus to Minus
a * b    // Plus to Times
a / b    // Plus to Divide

Comparison Operators

// Original: a > b
a >= b   // Greater to GreaterOrEqual
a < b    // Greater to Less
a <= b   // Greater to LessOrEqual
a == b   // Greater to Equal

Logical Operators

// Original: a && b
a || b   // And to Or

// Original: !a
a        // Negate removal

Boundary Mutations

// Original: i < 10
i <= 10  // Less to LessOrEqual
i < 11   // Boundary change
i < 9    // Boundary change

Return Value Mutations

// Original: return value
return undefined;  // Remove return
return !value;     // Negate return
return "";         // Empty string
return 0;          // Zero
return null;       // Null

Understanding Results

Mutation States

State	Description	Action
Killed	Test failed = mutation caught	Good!
Survived	Tests passed = mutation missed	Add tests
Timeout	Tests took too long	Check infinite loops
No Coverage	No tests cover this code	Add tests
Compile Error	Mutation broke compilation	Ignore

Mutation Score

Mutation Score = (Killed / Total) * 100%

• 80%+: Excellent test quality
• 60-80%: Good, room for improvement
• 40-60%: Weak tests, many gaps
• <40%: Critical test deficiency

Improving Mutation Score

1. Boundary Testing

// Weak: Only tests middle values
it('validates age', () => {
  expect(isValidAge(25)).toBe(true);
});

// Strong: Tests boundaries
it('validates age boundaries', () => {
  expect(isValidAge(0)).toBe(true);    // Min boundary
  expect(isValidAge(-1)).toBe(false);  // Below min
  expect(isValidAge(150)).toBe(true);  // Max boundary
  expect(isValidAge(151)).toBe(false); // Above max
});

2. Condition Coverage

// Original
function process(a, b) {
  if (a > 0 && b > 0) {
    return 'both positive';
  }
  return 'not both positive';
}

// Weak: Only tests one path
it('processes positive', () => {
  expect(process(1, 1)).toBe('both positive');
});

// Strong: Tests all conditions
it('processes various combinations', () => {
  expect(process(1, 1)).toBe('both positive');
  expect(process(-1, 1)).toBe('not both positive'); // a negative
  expect(process(1, -1)).toBe('not both positive'); // b negative
  expect(process(0, 1)).toBe('not both positive');  // a zero
});

3. Return Value Testing

// Weak: Only tests truthy
it('checks admin', () => {
  expect(isAdmin(adminUser)).toBeTruthy();
});

// Strong: Tests exact values
it('checks admin status', () => {
  expect(isAdmin(adminUser)).toBe(true);
  expect(isAdmin(regularUser)).toBe(false);
  expect(isAdmin(null)).toBe(false);
});

Incremental Mutation Testing

For large codebases, run mutations on changed files only:

# Only mutate changed files
git diff --name-only origin/main | xargs npx stryker run --mutate

CI Integration

GitHub Actions

name: Mutation Testing

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - run: npm ci

      - name: Run Mutation Tests
        run: npm run test:mutation

      - name: Upload Report
        uses: actions/upload-artifact@v4
        with:
          name: mutation-report
          path: reports/mutation/

Performance Optimization

Reduce Mutation Scope

{
  "mutate": [
    "src/core/**/*.js",
    "!src/core/**/*.test.js",
    "!src/core/generated/**"
  ]
}

Increase Parallelism

{
  "concurrency": 8,
  "testRunner": "vitest"
}

Filter Mutators

{
  "mutator": {
    "excludedMutations": [
      "StringLiteral",
      "ObjectLiteral"
    ]
  }
}

When to Use

Good Candidates

• Critical business logic
• Security-sensitive code
• Mathematical calculations
• State machines
• Validation logic

When to Skip

• Generated code
• Configuration files
• Third-party wrappers
• UI components
• Test utilities

Anti-Patterns

1. Chasing 100%: Diminishing returns above 90%
2. Ignoring Timeouts: Fix infinite loop mutations
3. Testing Everything: Focus on critical paths
4. No Baseline: Establish baseline before improving
5. Infrequent Runs: Run on every PR