| name | test-strategy |
| description | Coverage-design method for qa-engineer — pyramid ratios per archetype, equivalence/boundary/property case selection, mutation score as the real coverage signal, and a flake-quarantine policy. Turns "coverage is 90%" (a number with no method) into a defensible test plan. Emits TEST-STRATEGY-{slug}.md, which the QA gate checks exists. |
| when_to_use | Apply when:
- qa-engineer is planning tests for a medium/large feature (>1 file, real behaviour risk)
- a coverage % is being reported and you need to know whether it MEANS anything
- deciding what to test (not just measuring what's covered)
Do NOT apply to:
- nano / tiny changes (typo, rename) — overhead exceeds value
- pure docs changes
|
| effort | medium |
| allowed-tools | Read, Write, Grep, Glob, Bash |
| paths | ["tests/**","docs/qa-reports/**"] |
test-strategy
Coverage % is a number with no method. This skill makes the QA plan defensible:
what to test, why that set, and a real signal that the tests bite.
qa-engineer emits docs/qa-reports/TEST-STRATEGY-{slug}.md capturing the four
decisions below; the QA gate checks the file exists for medium/large features.
1. Pyramid ratio — pick by archetype (not by reflex)
The right unit:integration:e2e mix depends on where the risk lives.
| Archetype | unit | integration | e2e | why |
|---|
| library / devtools / cli | 80% | 15% | 5% | logic-dense, few I/O seams |
| web-app / saas | 60% | 30% | 10% | request→db→render seams dominate |
| commerce / fintech / marketplace | 50% | 35% | 15% | money paths need cross-component proof |
| data-platform / streaming | 45% | 45% | 10% | correctness lives in pipelines, not units |
| ai-system / agent-product | 50% | 20% | 10% + evals 20% | behaviour is the contract → eval set (see [[decision-eval]]) |
State the chosen ratio in TEST-STRATEGY and justify any deviation.
2. Case selection — equivalence / boundary / property
Don't enumerate inputs; partition them.
- Equivalence classes — one representative per class of behaviour (valid, invalid, empty, max). Testing 5 valid ids ≠ testing 5 classes.
- Boundaries — the bug lives at the edge: 0, 1, n-1, n, n+1, off-by-one, empty, overflow, the threshold itself.
- Property-based — for pure/transform logic, assert invariants over generated input (round-trip
decode(encode(x))===x, idempotency, ordering) instead of hand-picked cases. Use fast-check / Hypothesis where the logic warrants it.
3. Mutation score — the only coverage that proves the tests bite
Line/branch coverage proves code ran, not that a test would fail if it broke.
Mutation testing (Stryker / mutmut / cargo-mutants) flips operators/conditions
and checks a test catches it. A "90% covered" module with 30% mutation score has
assertion-free tests. Target: mutation score ≥ 60% on changed logic-dense files;
report it in TEST-STRATEGY when the change is logic-heavy. Coverage % is the cheap
proxy; mutation score is the truth.
4. Flake quarantine — a flaky test is worse than no test
A test that fails ~randomly trains the team to ignore red. Policy:
- A test that fails non-deterministically is quarantined (skipped + tracked in a Beads bug), not left to rot the signal.
- Quarantine is time-boxed: a bug with an owner and a deadline, not
it.skip forever.
- Root-cause before re-enabling (shared state, timing, real network, ordering).
- This is the same discipline the eval runner needs (see the eval
flaky flag in
tests/eval/runner.mjs — stddev > 0.1 is a flake signal).
Output: TEST-STRATEGY-{slug}.md
# TEST-STRATEGY-{slug}
- Archetype + chosen pyramid ratio (+ justification for any deviation)
- Equivalence classes & boundaries enumerated per changed unit
- Property-based candidates (which logic, which invariants)
- Mutation-score target + result (for logic-dense changes)
- Quarantine list (flaky tests + their tracking bug)
Done = the plan answers "why THIS set of tests", not just "how much is covered".
