soak-test-classification

name	soak-test-classification
description	Use when summarizing 24h+ soak, chaos, fault-injection, or long-running addon test results. Classifies each fault into invariant-break, product-path-failure, harness-race, or external-environmental-cascade before declaring ACCEPTED / blocked, avoiding false "all pass" or false product-bug reports.
allowed-tools	Bash(rg *) Bash(grep *) Read

Soak Test Classification

Hard Rules

1. Fault count is not the verdict. A run with faults may still be accepted if no product invariant broke and all faults are classified.
2. Classify each fault by first failing layer. Do not collapse product, harness, and environment failures into one pass/fail number.
3. Use four states only:
   • invariant-break
   • product-path-failure
   • harness-race
   • external-environmental-cascade
4. N=1 cannot prove harness-race or cascade. Need at least one same-class control sample or repeat that shows product path can succeed.
5. ACCEPTED means invariant-break = 0 and product-path-failure = 0. Harness and environment findings become follow-up tickets, not silent ignores.
6. Document caveats. Accepted-with-caveats is valid; hiding caveats is not.

When To Invoke

Use this skill when:

• Reporting a 24h soak / chaos / fault-injection cycle.
• Seeing a long run with mixed PASS/FAIL/fault records.
• Deciding whether a run blocks shipping.
• Reviewing someone else's "all tests passed" or "product failed" summary.
• Separating product failures from runner races or external environment cascades.

Four-State Classification

State	Meaning	Blocks Ship?
invariant-break	Data, acknowledged writes, role safety, or durability invariant broke	Yes
product-path-failure	Product recovery path failed after a valid fault	Yes
harness-race	Test tool ordering/race caused the fault report	No, but fix harness
external-environmental-cascade	Outside environment caused the outlier	No, but document and fix env

Decision Flow

For each fault:

1. Did a product invariant break?
   • data loss, stale acknowledged write, dual-primary accepting writes, wrong durable state
   • yes -> invariant-break
2. Did the product recovery path fail with env/runner/probe clean?
   • yes -> product-path-failure
3. Is there evidence the test harness raced the product?
   • requires same-class control or repeated sample
   • yes -> harness-race
4. Is there evidence of outside environment cascade?
   • requires baseline/control showing product path normally completes
   • yes -> external-environmental-cascade
5. If none of the above is proven, mark unclassified and do not declare final ACCEPTED.

Minimum Evidence

Classification	Minimum Evidence
invariant-break	Direct invariant evidence, even N=1
product-path-failure	Clean env/runner/probe plus product recovery failure; prefer N>=2 before broad claims
harness-race	Failing sample plus same-class control sample where product succeeds
external-environmental-cascade	External event correlation plus baseline/control within expected window

Report Template

Soak result:
- Duration:
- Fault injections:
- invariant-break:
- product-path-failure:
- harness-race:
- external-environmental-cascade:
- unclassified:

Verdict:
- ACCEPTED only if invariant-break=0, product-path-failure=0, unclassified=0

Caveats / follow-up:
- harness fixes:
- environment fixes:
- product bugs:

Bad Patterns

Pattern	Why It Is Wrong	Fix
"All passed" with harness faults hidden	Future teams cannot reproduce confidence	Report accepted with caveats
"Product failed" from one timeout	N=1 and may be harness/env	Classify first failing layer
Treating external cascade as pass	It may hide test infrastructure risk	Track env follow-up
Ignoring unclassified faults	Unknowns become silent risk	Keep verdict open

Related Docs

• docs/addon-soak-test-result-classification-guide.md
• docs/addon-test-acceptance-and-first-blocker-guide.md
• docs/addon-evidence-discipline-guide.md