testing-strategy

name

testing-strategy

description

Design a test suite for an open-source project that contributors can actually run and trust. Use when a project has no tests, when the suite is slow or flaky, when deciding what to test and at what level, when setting a coverage policy, or when contributors' PRs keep failing CI for unrelated reasons. Covers the test pyramid, fixtures, property-based and snapshot testing, cross-version matrices, flake elimination, and making tests runnable in one command. Also use for "how do I test this" and for reviewing test quality in a PR.

Testing Strategy

For an open-source project, the test suite has a second job beyond correctness: it is the contract that lets strangers change your code safely. Without it, every external PR is a risk you must personally verify, and you become the bottleneck.

The one-command rule

A new contributor must be able to run the full suite in one command, from a fresh clone, with no tribal knowledge:

make test          # or: npm test / pytest / cargo test / go test ./...

Requirements, all of them non-negotiable:

• No manual database setup. Use containers, or a file-backed engine in tests.
• No secrets required. Tests needing credentials are skipped by default with a clear message, and run only in CI with secrets present.
• No network by default. A suite that fails on a plane fails in a corporate proxy too, and looks like your bug.
• Deterministic. Seed randomness, freeze time, sort collections before comparing.
• Fast enough to run before every commit. Target under 60 seconds for the default suite; push the slow tests behind a flag.

If setup takes more than git clone && make test, contributions drop measurably. This is the highest-leverage testing investment an OSS project can make.

The pyramid, and where projects get it wrong

Level	Share	Runtime	Tests
Unit	~70%	ms	Pure logic, edge cases, error paths
Integration	~20%	~s	Component boundaries, real DB/filesystem
End-to-end	~10%	~10s+	The two or three flows that must never break

Two common failure shapes:

• Ice cream cone (mostly E2E) — slow, flaky, and when it fails it doesn't tell you where. Symptom: "just re-run CI" is normal team advice.
• Hourglass (units + E2E, no integration) — every component works, the assembly doesn't. Symptom: bugs are always at the seams.

Test the public API, not internals. Tests bound to private functions turn every refactor into a test rewrite, which teaches contributors that tests are an obstacle.

What to actually test

Prioritize by consequence-of-failure, not by ease:

1. The documented behavior. Every promise in the README should have a test. This catches the most damaging class of bug — the one that makes your docs a lie.
2. Error paths. Most projects test the happy path and ship broken error handling, which users hit precisely when they are already frustrated.
3. Boundaries. Empty, one, many, max. Zero, negative, off-by-one. Unicode, emoji, RTL text, embedded newlines. Paths with spaces. Windows line endings.
4. Every bug you fix. A regression test at fix time is the cheapest test you will ever write, and it names its own reason for existing.
5. The public API shape itself — snapshot the exported surface so accidental breaking changes fail CI (see api-design).

Do not test: framework behavior, third-party libraries, getters and setters, or anything whose test is a restatement of the implementation.

Techniques worth the investment

Table-driven tests — the highest density of coverage per line of test code, and the easiest for contributors to extend when they find a new case.

tests := []struct{ name, in, want string }{
    {"empty", "", ""},
    {"unicode", "café", "cafe"},
    {"emoji", "a👍b", "ab"},
}
for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) { /* ... */ })
}

Property-based testing (Hypothesis, fast-check, proptest, QuickCheck) — for parsers, serializers, encoders, and anything with an inverse. One property replaces a hundred examples and finds the case you would never have written:

@given(st.text())
def test_roundtrip(s):
    assert decode(encode(s)) == s

Snapshot tests — good for CLI output, generated code, and rendered markup. Two rules or they rot: review every snapshot diff as if it were source code, and never run --update-snapshots without reading the diff. An auto-approved snapshot suite tests nothing.

Golden files for compilers, formatters, and codegen: input file → expected output file, both in the repo, regenerable with one flag.

Mutation testing (mutmut, Stryker, cargo-mutants) — the honest answer to "is our coverage real". Expensive; run it quarterly or in a nightly job, not per-PR.

Coverage policy

Coverage is a diagnostic, not a target. Enforcing 100% produces tests written to satisfy the instrument.

A policy that works in practice:

• Track coverage; do not gate on the absolute number. Publish it, watch the trend.
• Gate on coverage of the diff — new code should come with tests. Codecov and similar report this natively; it is a much fairer bar for contributors than a global percentage they cannot influence.
• Exempt generated code, __main__ blocks, and platform-specific branches in config, so the number means something.
• Uncovered error paths are the signal that matters. If your happy path is at 95% and error handling is at 20%, coverage tooling is telling you exactly what to do.

Cross-version and cross-platform

Open-source code runs where you do not. Decide the support matrix explicitly, publish it, and test it — an untested claim of Windows support is worse than no claim.

strategy:
  fail-fast: false
  matrix:
    os: [ubuntu-latest, macos-latest, windows-latest]
    version: ['3.10', '3.13']          # oldest supported + newest
    include:
      - { os: ubuntu-latest, version: '3.11' }   # fill the middle on one OS only

fail-fast: false is important: you want to see all failing cells, not the first. Full cartesian matrices waste CI minutes — test the corners exhaustively and the middle on one platform. See ci-pipelines.

Windows deserves specific attention because it is where cross-platform bugs live: path separators, CRLF vs LF, case-insensitive filesystems, PATH length limits, and no fork().

Killing flakes

A flaky test is worse than no test: it trains everyone, including you, to ignore red CI. Treat a flake as a P1 bug.

Root causes, in the order you should suspect them:

1. Time — sleep() as synchronization, tests that fail near midnight or DST boundaries, timeouts tuned to a fast laptop. Fix: inject a clock; poll for a condition with a generous timeout instead of sleeping a fixed duration.
2. Order dependence — shared global state, leaked singletons, a database not reset between tests. Fix: run with --shuffle/-p random ordering in CI so order dependence fails immediately rather than mysteriously.
3. Concurrency — real races in the code under test. These are the valuable ones; the test is telling you the truth.
4. Network and external services — never in unit tests; stub at the boundary.
5. Resource contention — ports, temp file names, CPU starvation on shared runners. Fix: bind to port 0 and read back the assigned port; use unique temp dirs.

Process: quarantine the flake into a separate job immediately so it stops blocking contributors, open an issue with the failing logs, and fix or delete it within a sprint. A permanently quarantined test is a deleted test with extra steps — be honest and delete it.

Contributor-facing test docs

In CONTRIBUTING.md, concretely:

## Tests
make test            # full suite (~40s)
make test-unit       # fast, no containers
make test-e2e        # requires Docker
pytest tests/test_parser.py::test_unicode -x   # single test

Also state: what a new feature needs (tests + docs), what a bug fix needs (a regression test), and what happens when CI is red for unrelated reasons — because a contributor who cannot tell "my fault" from "not my fault" usually just leaves.

Anti-patterns

• Tests requiring undocumented local setup. The contribution killer.
• sleep(2) as synchronization. Slow and flaky.
• Asserting on log output for behavior that has a return value.
• One test asserting fifteen things. When it fails you learn nothing.
• Mocking the thing under test. You are testing your mock.
• Mocks so deep the test encodes the implementation. Any refactor breaks it.
• Skipped tests with no issue link. Delete them or fix them.
• if (CI) skip(). The test is a lie in exactly the environment that matters.
• Coverage gates that reject a one-line docs-adjacent fix. You just taught a first-time contributor that this project is not worth the trouble.