writing-tests

name

writing-tests

description

How to write, change, or review tests in this repo — the load-bearing rule is test real behavior, not source text. Also covers non-vacuity (red-on-old → green-on-new), covering enumerated sets member-by-member, fuzzing parser/Unicode logic, hostile-pre-state filesystem tests, and the c8 / kcov / mutation coverage floors. Activate whenever the user asks to write, add, fix, refactor, strengthen, or review tests ("write a test", "add tests", "test this", "regression test", "cover this", "why didn't this test catch it"), or when a coding task's last step is testing the change you just made.

Writing Tests

A test earns its place by going red the moment a real defect returns — not by executing lines for a coverage number. A test that stays green against a broken implementation guards nothing; every rule below serves that.

Test behavior, not source text

Never grep the implementation as a proxy for what it does. assert "chmod 0444" in script passes whether or not that line ever runs, breaks on a harmless rename, and stays green against a broken implementation that still contains the token. Instead drive the real code path and assert the observable outcome: run the function or script under stubs (a fake install, docker, node, id on PATH that records how it was called) and check what actually happened — the file written and its mode/owner, the argv+env a binary was exec'd with, the exit code, the bytes on stdout/stderr, the state left on disk.

The only legitimate exceptions parse data, not imperative logic: validating a config artifact the product ships and loads at runtime (a settings.json, an execpolicy.rules) with its real parser, or pinning a cross-file structural contract two files must agree on when no single process can observe both. A grep of imperative source to confirm it "does X" is never an exception.

Prove the test can fail (non-vacuity)

• A bug-fix test isn't done until you've watched it fail on the unfixed code. Revert the fix (or run against git show <base>:<file>); confirm it goes red for the bug's signature, not a setup/import error; restore; confirm green. State "red on old → green on new" in your self-critique.
• New code with no pre-fix counterpart: prove non-vacuity by mutating the new logic (flip a comparison, drop an emit, return the wrong status) and confirming a test catches it. CI's mutation gate does this for JS/Python — for bash the invert-and-run check is yours, every time.
• Assert the invariant a bug violated, not today's symptom, so the whole class can't recur by another path. When that principled test is hard to write, the difficulty usually points at a design smell.
• A fix's own comment is the spec its test must be driven from. When the fix (or the code it touches) claims a generality property in a comment — "matched on the phrase, not the exact wording, so a reworded release still routes here", "names EVERY path level", "any of these retryable phrasings" — that sentence is the behavior under test, and a test exercising one example of it is instance-pinned: it passes while a regression narrows the code back to the single case the comment promised to generalize past. Drive the test from the claim (every reworded variant, every path level, every alternation member), not from the one input that first triggered the bug. A comment that promises more generality than its test exercises is the reviewer's cheapest tell for a hollow regression test.

Would a generalizable invariant have caught it in advance?

For every bug you fix, before writing the instance regression, ask: what property of any correct run did this bug violate — and can a test assert that property without knowing today's symptom? The instance test ("this input now returns C") guards one input; the invariant guards the whole class, including the next variant that reaches the same fault by another path. Reach for the invariant first; fall back to the instance test only when the invariant genuinely can't localize the regression.

The bugs that most need this are silent degradations — the system emitted a plausible-looking value (a —, a small cost, a token count) that was wrong and nothing failed. A per-input test can't catch what it never thought to check; a property test can. Three recurring shapes, and the invariant that catches each with no foreknowledge:

• Completeness / reachability — an output the report promises is never produced (a dashboard column scored only under a config the run never includes reads as "no data yet"). Invariant: every promised output is emitted by at least one cell of a complete run, and "unmeasured" is a distinct sentinel from "empty".
• Conservation / monotonicity — an input the system spends is never counted (a cost function that ignores a token category undercounts silently). Invariant: the total is strictly monotonic in every billable field — bump each field, assert the total moves — so a category the accountant forgets reds.
• Economic asymmetry — a resource whose cost model is lopsided (write ≫ read) is used on its expensive side. Invariant: in steady state the cheap operation dominates, and a run that inverts the ratio fails.

When the invariant is hard to state, that difficulty usually names a design smell (a metric that can't be reached, a spend with no single accounting point) — fixing the design beats papering over it with one more example test.

Cover the whole input domain

• An enumerated set (regex class, |-alternation, lookup table, allowlist) needs a case per member, not per line — 100% coverage fires the construct on one input and leaves the rest unverified, and a dropped member is invisible to coverage and often to mutation. Drive cases from the SSOT list (for...of a JS array so a new member without a case fails to compile).
• Parser / Unicode / character-property logic gets a fast-check property test over its real domain (joiner scripts, emoji parts, every invisible class, lone surrogates), pinning invariants — idempotence, output-is-a-subsequence, never-throws — not just ASCII examples.
• Code that writes into a user's filesystem gets driven through hostile pre-states — missing, regular file, directory, valid symlink, dangling symlink, wrong permissions — each asserting a well-defined outcome (post-condition met or a clear failure), never a silent exit-0 or a leaked raw cp:/ln: error. Test the reinstall-over-stale-state re-run too.
• Redaction-test needles must be credential-shaped, never low-entropy — the engine skips placeholder-looking values (repeated-char runs, CAPS_WITH_UNDERSCORES metavariables, changeme), so a low-entropy fixture silently asserts nothing. Use the canonical shared needle from tests/secret-format-samples.json.

Cross-platform

Host-facing code (setup.bash, bin/claude*, uninstall) hits BSD coreutils on macOS, so mark its tests @pytest.mark.cross_platform — the cross-platform-tests.yaml matrix then runs them on Linux and macOS; marking is all it takes. Keep them OS-agnostic (shutil.which, tmp_path, no GNU-only flags). Container/devcontainer e2e stays Linux-only. Interaction features get Playwright e2e (mobile + desktop).

A cross_platform test is untimed until a main run publishes the refreshed durations map to R2 (the pytest gate's upload step — the map is never committed), so the cost-aware sharding (tests/_sharding.py) weights it at the p90 of known costs to keep it from clumping. That's still only an estimate: the WSL2 DrvFs legs run ~4× slower (FS/process ops cross the 9P bridge), and a subprocess/FS-heavy new test is slower there than its p90 estimate — so it can overload one DrvFs shard and blow its timeout-minutes. If a shard times out or runs far longer than its siblings, that is a load-balance bug to fix at the root (the test is too slow/subprocess-heavy for a host test, or the durations/shard sizing need refreshing) — never a flake to re-run past. Prefer in-process assertions over spawning python3/jq/git per case when a cross_platform test can express the same behavior, precisely because subprocess spawns are what the DrvFs bridge taxes.

Keep the suite honest and readable

• Never skip or weaken a test unless asked — including silently dropping an assertion while refactoring. When you parametrize, every assertion and distinct input from the originals must survive (case-specific checks become per-case params).
• Shared stubs/fixtures live in conftest.py/_helpers.py and are imported, never re-pasted as a second source of truth.
• Prefer exact-equality assertions; parametrize only when it shortens the code and stays readable.
• Prefer a real SSOT over a drift guard — make a duplicated value-set structurally unable to diverge (import the shared module, or generate the JSON from it) rather than testing that two copies still match.
• Don't gate a load-bearing test behind skipif(which(tool) is None) — the tool is missing precisely in CI, so the gate becomes a silent no-op there. Install the tool in the job and let a missing binary error.

Does the health check need a new entry?

When your change adds a runtime prerequisite, a new failure mode, or a state a user could land in and not know why a launch degrades, ask whether glovebox doctor (bin/glovebox-doctor) needs a matching check — and if so, add it in the same change. A test proves the code works; a doctor entry is how a user discovers it's broken on their machine before a launch does it for them. Triggers: a new required CLI/daemon/device, a new credential or sign-in the launch depends on, a new symlink or service, a new "silently degraded to host execution" path. The doctor is read-only and must skip (not fail) when an optional tool is absent, but a genuinely required prerequisite that's missing is a fail with the exact fix command. This is easy to forget precisely because the feature works on the machine you built it on — treat "did I owe a doctor check?" as a standing item on every feature/fix, not something to remember ad hoc.

Before pushing: fast and targeted only

Never run full suites locally — that's CI's job. Run only the targeted fast tests for the files you changed and their obvious neighbors: .venv/bin/pytest tests/test_foo.py -n auto -q -p no:cacheprovider, narrowed with -k, stopped early with -x, re-running failures with --lf. The full sweeps (pytest tests/, pnpm test, c8, kcov, mutation) are minutes-slow and belong to CI.

Coverage floors

100% is enforced per file, and must be met with representative behavioral tests, not a test per residual branch — a branch that only a hollow test can reach is usually code worth simplifying (or mark it # kcov-ignore-line with a justification).

• JS hooks (.claude/hooks/*.mjs): c8, 100% lines/branches/functions (pnpm test:coverage).
• Bash wrappers (bin/claude*): enrolled in kcov (tests/_kcov.py), 100% real lines. Cover a new module with in-process tests — subprocess calls don't trace into the child interpreter.
• Mutation proves the assertions bite; it runs on CI only — never run Stryker/cosmic-ray in this sandbox (see CLAUDE.md → Self-Critique Loop).

Cover every new file, and never opt one out silently. Adding a file to a coverage exclusion is a real loss of a safety net — surface the argument and get sign-off first, and you still owe a behavioral test of the invariant that mattered. Enrollment mechanics and kcov blind spots (empty case arms, /dev/kvm probes) live in .claude/dev-notes.

One example

Weak (source grep): assert "install -o root -g root -m 0444" in entrypoint_src — green even if the install never runs, red on a harmless reformat.

Strong (behavioral): put a fake install on PATH that records mode owner group src dest to a capture file, run the real bring-up, then assert the captured record for the config path is 0444 root root — including the re-run-over-an-existing-file pre-state. It fails exactly when the mode/owner regresses, and survives a rename.