| name | unit-testing-rubric |
| description | Standards for writing, reviewing, and evaluating unit tests. Use whenever writing new tests, reviewing existing tests, deciding what to test, assigning a test to a fast vs. full feedback loop, or diagnosing flaky, slow, or low-value tests. Covers structural fidelity, the FIRST properties, behavioral invariants, and coverage-as-diagnostic. |
Unit Testing Rubric
Purpose
Create a trustworthy, deterministic, affordable test suite that protects
behavioral invariants, supports refactoring, and provides rapid feedback.
Flaky tests are defects in the feedback system and must be fixed, replaced, or
removed.
Testing Integrity
1. Fidelity Rule — Tests must be structurally honest. The production
behavior under test must execute the same way in the test harness as in
production. Introduce seams around dependencies, not around behavior.
Would production execute the same behavior if this test did not exist?
2. Documentation Rule — A microtest is executable documentation. A developer
should understand the behavior, context, action, expected result, and important
edge cases without reading the implementation.
Can a developer understand the behavior from the test alone?
3. Single Action Rule — An Arrange–Act–Assert test performs exactly one
action in one context. Multiple assertions are acceptable when they examine
evidence from the same action.
What is the single action being performed?
FIRST Properties
4. Fast Rule — Tests should support rapid feedback. The fast suite (however
the project invokes it) should complete in well under a minute; treat ~30
seconds as a reasonable default budget absent a project-specific one.
Is this test worth spending part of the fast-feedback budget?
5. Isolation Rule — Tests must not depend on execution order, shared mutable
state, hidden fixtures, or environmental assumptions.
Can this test pass independently on any machine?
6. Repeatability Rule — Accepted tests must be deterministic. Control time,
randomness, IDs, concurrency, and external dependencies.
Will this test produce the same result every time?
7. Self-Verifying Rule — Tests produce an automatic, unambiguous pass/fail
result.
Does the test clearly determine success or failure without human
interpretation?
8. Timely Rule — Tests are written alongside behavior, protect regressions,
and evolve with the product.
Is this test protecting currently desired behavior?
Behavioral Quality
9. Behavioral Invariant Rule — Every permanent test specifies a behavior
that should remain true until the product no longer wants it. Tests exist to
lock down behavioral invariants.
What invariant does this test protect?
10. Potency Rule — A test must fail for a plausible violation of the
invariant it specifies. A test that cannot detect a realistic defect provides
little value.
What bug would make this test fail?
11. Failure Clarity Rule — When a test fails, it should clearly illuminate
the violated invariant and help localize the problem. Failures should be
specific, local, and diagnostic.
If this test fails, will a developer know what invariant was violated?
Portfolio Quality
12. Affordability Rule — Confidence should be obtained at the cheapest
reliable test level. Prefer narrow deterministic tests over expensive broad
tests.
Is there a cheaper deterministic test that provides the same confidence?
13. Feedback Loop Rule — Assign each test to the earliest useful feedback
loop: a fast loop (e.g. run_tests) for deterministic feedback on every change,
and a full/slow loop (e.g. full_test) for additional deterministic confidence
that doesn't belong in the fast one (end-to-end, slow, or integration-heavy
cases) — name the two loops whatever the project calls them. Most confidence
should live in the fast loop.
Why doesn't this confidence belong in the earlier feedback loop?
14. Coverage Diagnostic Rule — Coverage is a diagnostic signal, not the
goal. Use it to discover unprotected decisions, transformations, and invariants.
Do not create tests merely to increase coverage percentages.
What behavioral invariant is missing behind this uncovered code?
Lifecycle
15. Prophet / Guide / Guard Rule — Every permanent test serves three
purposes: Prophet (specifies a future invariant before implementation),
Guide (provides the target behavior during implementation), Guard
(protects behavior during maintenance and refactoring).
Does this test successfully act as Prophet, Guide, and Guard?
Operational Expectations
- All accepted tests are deterministic.
- No retries, quarantines, reruns-until-pass, or tolerated failures.
- Prefer behavior assertions over implementation assertions.
- JSON object field order is not asserted unless contractually significant.
- Most confidence should come from the fast loop.
- Every test should justify its value against its cost.
- Refactor tests to reduce fragility, redundancy, and cost while preserving
invariants.
- Delete tests only when the behavior is no longer desired, or when the test is
redundant, flaky, or provides no unique confidence.
Common Pitfalls
Ambiguous fixture selection. When a test needs to pick one item out of
several candidates that share a common identifying field (e.g. two functions,
signatures, or records with the same name/owner but a different shape),
filtering on that one shared field alone can silently bind to the wrong
candidate — the test then validates the wrong thing while still appearing to
pass. Filter on every field that actually distinguishes the candidates, not
just the most obvious one.
Widening behavior without breaking coverage. When an existing function's
behavior is being deliberately widened (recognizing more inputs, covering more
cases) without changing its signature, audit every existing test that asserts
on it before changing the implementation, to confirm none assumes the
narrower behavior. A widening change is verifiably non-breaking when no
existing assertion could contradict the new, broader behavior — don't rely on
a regression to reveal a conflict you could have checked for directly.
