testing-strategy

name	testing-strategy
description	Use when the user asks "how should I test this", "what tests should I write", "is this enough test coverage", "how do I test this without mocking everything", "TDD vs writing tests after", or discusses unit vs integration vs E2E tests. Provides senior engineering guidance on building effective test suites.

Testing Strategy

The test pyramid is a ratio, not a rule. The goal of a test suite is confidence that the system behaves correctly — the cheapest possible confidence, measured in CI time and maintenance burden. A test suite that takes 45 minutes to run is not a safety net; it is a toll booth.

When to Activate

• Designing a test suite for a new service or feature
• Evaluating existing test coverage
• Deciding between mocking and integration testing
• Establishing team testing standards
• Deciding whether to practice TDD

The Test Pyramid

        /\
       /E2E\         Few — slow, brittle, high confidence
      /------\
     /Integr. \      Some — moderate speed, high confidence on integration
    /----------\
   /   Unit     \    Many — fast, cheap, high confidence on logic
  /--------------\

Unit tests: test a single function or class in isolation. Fast (milliseconds), deterministic, cheap to write and maintain. Best for: pure logic, algorithms, edge cases, parsing.

Integration tests: test multiple components working together — your code + a real database, your code + a real HTTP client. Slower (seconds), but prove that the pieces actually fit. Best for: data access, external service integration, message queues.

E2E tests: exercise the system through its public interface as a real user would. Slowest (minutes), most brittle, highest confidence. Best for: critical user journeys, smoke testing after deployment.

The Anti-Pyramid (and why it fails)

A test suite that is 80% E2E tests:

• Takes 20+ minutes to run → developers stop running it locally
• Fails due to environment issues, not code issues → false negatives → developers ignore failures
• Gives no signal about where a failure is → slow debugging

A test suite that is 80% unit tests with heavy mocking:

• Passes even when integration is broken
• Mocks drift from the real behavior of the thing they mock
• Provides false confidence

The right ratio depends on risk profile. Start with the pyramid shape and adjust based on evidence.

What to Test

Test Behavior, Not Implementation

A test that asserts on internal state (private methods, internal data structures) breaks every time the implementation changes, even when behavior is preserved. Test the observable behavior — what goes in, what comes out, what side effects occur.

Wrong: test that _parse_rows is called with the right arguments Right: test that given input X, the output is Y

Test the Contract

For every public function:

1. The happy path (expected input → expected output)
2. Edge cases (empty input, boundary values, maximum values)
3. Error cases (invalid input → correct error type/message)
4. Side effects (if the function writes to the database, test that it wrote correctly)

What Not to Test

• Third-party libraries (test that you're calling them correctly, not that they work)
• Generated code
• Simple getters/setters with no logic
• Framework scaffolding

Testing these adds maintenance burden with no confidence gain.

Integration Testing Without Mocks

The most common mistake in integration testing is mocking the database, the queue, or the HTTP client — then writing unit tests and calling them integration tests. This defeats the purpose.

Test with Real Dependencies

Use Docker Compose or Testcontainers to spin up real dependencies in CI:

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: test
  redis:
    image: redis:7

The test starts a real Postgres, runs against it, and tears it down. The test now proves the SQL is correct, the schema matches expectations, and the ORM is configured right.

When Mocking Is Appropriate

Mock external dependencies you do not control and cannot run locally:

• Third-party payment providers
• SMS/email delivery services
• External partner APIs

Mock at the boundary — the HTTP client, not your internal service code. Use recorded responses (VCR cassettes) where possible to keep mocks anchored to real behavior.

TDD (Test-Driven Development)

TDD is a design tool, not just a testing technique. Writing the test first forces you to define the interface before the implementation.

The TDD loop:

1. Write a failing test that describes the desired behavior
2. Write the minimal implementation to make it pass
3. Refactor the implementation while keeping the test green

When TDD Works Well

• Pure logic: parsers, algorithms, data transformations
• Well-defined requirements with known inputs/outputs
• Refactoring with a safety net

When TDD Is Harder

• Exploratory development (you don't know the interface yet)
• UI-heavy code (rendering, layout)
• Code with complex external dependencies

TDD is not dogma. The obligation is that every behavior is covered by a test — the order in which you write the test and the code is secondary.

Test Quality

A Good Test Is

• Readable: another engineer understands what it tests without reading the implementation
• Isolated: does not depend on other tests or external state left by other tests
• Deterministic: always produces the same result given the same code
• Fast: does not sleep or do unnecessary I/O
• Testing one thing: when it fails, the failure message points to the problem

Test Naming

Name tests to describe the scenario and expected outcome:

test_create_order_returns_201_when_input_is_valid
test_create_order_returns_400_when_email_is_missing
test_create_order_deduplicates_on_idempotency_key

A test name that says test_create_order tells you nothing when it fails.

Flaky Tests

A flaky test (one that passes sometimes and fails sometimes without code changes) is worse than no test — it trains engineers to ignore failures. Treat flakiness as a P1:

1. Quarantine the test immediately so it doesn't pollute CI
2. Investigate root cause: shared state, time dependency, network dependency, race condition
3. Fix and restore

Code Coverage

Coverage is a floor check, not a confidence metric. 80% line coverage means 20% of lines are untested — it says nothing about whether the tests are meaningful. Chase coverage for gaps, not as a target.

Gotchas

1. Mocking your own code: mocking the class you wrote to test the class that calls it means you're testing the mock, not the integration. Use real implementations.

2. Tests that test the framework: if your test is just checking that Django's ORM saves records, you're testing Django. Test that your code calls the ORM correctly.

3. One test per function: some functions need 10 tests (edge cases). Some functions need none (trivial delegation). Tests per function is not the right metric.

4. Shared test fixtures with mutable state: test A mutates a shared fixture → test B sees unexpected state → test B fails intermittently depending on test order. Make fixtures explicit and scoped.

5. Testing error messages as strings: assert error.message == "User not found" breaks every time you improve the message. Assert on error type and code, not message text.

6. Slow tests that no one runs: a test suite that takes 20 minutes will be run in CI but not locally. Engineers merge without running tests. Invest in making tests fast — parallelize, reduce I/O, scope dependencies.

Integration

• code-review — evaluate test coverage and quality as part of every PR review
• debugging-methodology — failing tests are the entry point to the debugging loop
• api-design — contract tests validate API behavior across versions
• incident-response — regression tests prevent re-introduction of production bugs

References

• Test Patterns Reference — test double taxonomy, parameterized testing patterns, property-based testing

Skill Metadata

Created: 2026-04-10 Version: 1.0.0