write-tests

# Write Tests A good test fails **only** when real behavior breaks, and passes through every refactor or config change that preserves it. Most bad tests fail the opposite way: red on harmless changes, green while the real path is broken. Every rule below serves that one goal. ## Workflow: tracer bullets, not a batch 1. **Write ONE test at a time.** Assert first, watch it go red on the un-fixed code, make the code earn green, learn, then write the next. Never a batch up front: a batch written against *imagined* behavior pins what you guessed — those tests pass when the mechanism breaks and fail when it's fine. Each green cycle tells you what the next test should actually assert. 2. **Iterate on the fastest focused runner** (one file, one test name), and run the full suite only as a final gate before handing off. Check the exit code, not just the output — a runner that prints nothing and a green run look the same. On failures, read EVERY red test before fixing one; they often share a root cause. 3. **Before calling it done**, prove the test can fail (below) and walk the review checklist. ## What to assert - **Observable behavior through the outermost practical entry point** — return values, exit codes, persisted rows, HTTP responses, rendered output — never which internal functions ran or how a value is computed. A test on the public surface survives a rewrite of everything underneath; a test that reaches into internals breaks on every refactor and pins implementation, not behavior. Reserve isolated unit tests for genuinely tricky pure logic (parsers, schedulers, state machines). - **Nothing the compiler already guarantees.** A test that re-asserts a type signature — field shapes, rejected argument types — can only fail if the compiler failed first. Spend the budget on business rules, arithmetic, branching, ordering, edge cases, side effects. - **Actual values, not collection sizes.** For dedup/normalize/idempotency paths, `length == 1` passes even when normalization is broken; also assert the stored value equals the expected canonical form. - **The smallest scale that can show the behavior.** Two entities and one mechanism before crowds and integration; small tests fail fast with readable state and don't entangle five behaviors in one assert. - **The design contract, not current behavior.** When a test goes red, the reflex is to re-measure and pin the new number — resist it: a bar calibrated to whatever the code currently does silently encodes bugs as baseline. Write the assert from the stated contract and make the code earn it; if no contract exists, that's a question for the owner, not a number to measure-and-pin. Recalibrating is legitimate only when the contract itself changed. - **Deterministic claims tight, stochastic claims as distributions.** An invariant that holds every run gets an exact threshold. Anything noisy or tuning-dependent ("A usually beats B", "load balances evenly") must be asserted over a set of runs/seeds as a band — a single-sample pin on a stochastic outcome is not a weak test, it is a **blind** one: it certifies whatever the lucky sample did and can mask a systematic bias for months. If you must assert a noisy differential, widen the margin and name it chaos-marginal in a comment. ## Seams and mocks - **Don't couple tests to config — mock the seam.** A test keyed to a live config value (which flag is on, which tier is default) breaks when someone legitimately edits config. The fix is never `skipIf`/conditionals — a skipped assertion hides the coupling and stops covering the path. Feed the function fixed inputs, or stub the lookup so fixture ids resolve to fixed values. Litmus: "would this break if a config value changed with no logic change?" - **Don't over-mock — every mock is a frozen assumption.** Mock at the system's edges (network, clock, filesystem, third-party SDK, config lookup), never internal collaborators. Stubbing an internal hard-codes its current contract; refactor it and the test lies. Never make "was called with" the primary assertion when an observable outcome exists. Mocking three internal modules to test one function means: test one layer up, where they're real. - **Harnesses must wire the system the way production does.** If a bug only showed up against a real environment, the harness skipped an input production always sets — fix the harness, don't just fix the bug. ## Control variables and probes - **One variable per comparison.** Pin everything else — same seed, same counts, same configuration on both arms; disable mechanisms that aren't the subject. When a comparison test breaks, first ask whether an unrelated mechanism leaked into the experiment before touching the code under test. - **Validate, don't assume.** Never reason your way to a conclusion about *cause* — which path fired, where the failures come from — and act on it. Write a throwaway probe that reads public state and prints; the plausible story is wrong often enough to burn a session, and one probe redirects the whole effort. Probes are scratch: put them where they cannot be committed, delete them the moment the question is answered, or promote them into a real test if the behavior deserves a permanent guard. ## Tests run on someone's live machine A suite that grabs the desktop is a suite people stop running. Whatever the test needs, it takes the least intrusive form of it: - **Never take focus.** No activating the app, no `makeKeyAndOrderFront`, no raising, no moving the pointer, no simulated clicks into the session, no audio playback. A UI a person must not be interrupted by is still fully testable: build the window, drive its model, and assert on the rendered view. Where a screenshot is the evidence, capture the window's own image offscreen rather than photographing the screen. Keep activation on the code path a person triggers, and exercise it by calling the handler, not by launching the app repeatedly. - **Write nowhere the person keeps their own state.** Point every test at a scratch home, a scratch preferences domain or suite, and scratch install locations. A test must never modify the real config, library, login items or installed applications. Production may read an override the tests set; it must behave normally when that override is absent. - **Batch anything unavoidably visible.** If a check genuinely needs a real launch, do all of its observations in one run instead of one relaunch per assertion, and say in the evidence that it was visible. - **Leave nothing running.** Every process, window, server and temporary file the test started is stopped and removed, including on failure. ## Prove the test can fail A test you never saw fail is decoration. For any regression test — especially one written *after* the fix — falsify it once: revert or break the production code the way the bug would, confirm red *for the expected reason*, restore, confirm green. Verify the revert actually took: a stash or checkout with a wrong pathspec reverts nothing, silently, and the "red" run quietly tests the fixed code. The tell: the "red" numbers equal the green numbers. ## When a test goes red after a change Triage in order of likelihood before editing anything: 1. The test encodes **deleted semantics** — it pinned an accident of the old implementation. Re-spec it to the contract's real claim. 2. The experiment **lost control of a variable** — a new mechanism leaks in. Pin the variable. 3. The **margin was chaos-tight**. Widen it, with a comment saying so. 4. The code is **actually wrong**. Probe the state, find the mechanism. Never tune constants to make one test pass without rerunning the neighbors: coupled systems reshuffle. If two consecutive tweaks each break different tests, stop poking — the control surface is wrong; find the mechanism. ## Review checklist Walk this on any test diff, apply fixes in the same pass, re-run the suite: 1. Does an assertion restate a type guarantee? → delete it. 2. Would it break on a config edit with no logic change, or is it `skipIf`-conditional on config? → mock the seam. 3. Are assertions mostly mocked-internal "called with"? → test a layer up against observable output. 4. Does it pin a single sample of a stochastic outcome? → assert a distribution. 5. Does it assert a count without the value on a dedup/normalize path? → add the value assertion. 6. Is the function under test still called in production? → if the only callers are tests, delete the function and the test together. 7. Can it fail for the right reason? → falsify once to confirm. 8. Does it take focus, move the pointer, play sound, or write the person's real config, library or applications? → drive the model and a scratch location instead, and capture windows offscreen.