| name | test-strength |
| description | Test whether behavioral coverage can reject realistic wrong implementations. Use when tests are green but may be weak, after fixing a bug, and whenever code implements or changes a protocol, parser, serializer, state machine, lifecycle, event translation, browser interaction, adapter, or compatibility layer. Also use for falsification, mutation testing, property testing, differential testing, boundary integration, artifact verification, or flake detection. |
| compatibility | Requires runnable tests. Mutating production or test files requires user authorization and reversible isolation. |
Test strength
Execution is not protection. A strong test turns red when the guarded behavior is broken and exercises the layer where the claim lives.
1. Build the behavioral model
State the invariant, failure family, runtime boundary, and existing tests that claim protection.
For protocols, state machines, parsers, serializers, and event translation, write the dimensions before reading the implementation's branches. Derive them from each independent owner of behavior: specification or reference implementation, host/runtime semantics, and product contract. Include prior state, input or event, action or transition, negotiated modes or flags, relevant environment, and expected output plus post-state.
Enumerate every meaningful cell or partition it into documented equivalence classes. Mark impossible or excluded cells with the external rule that excludes them. Pairwise sampling is insufficient unless interactions between omitted dimensions are proved independent.
Complete when: the report contains the dimensions, values, exclusions, and the smallest wrong implementations the tests must reject. “Matrix run” without the table is incomplete.
2. Establish an independent oracle
Use a written specification, reference implementation, real producer or consumer, captured conformance vectors, or an independently written model. The production branch structure, constants copied from it, and expected values generated by the same helper are not independent.
For lifecycle events, validate fixture semantics against the real producer. Model each transition as pre-state + event → output + post-state. Do not reuse one mutable or contradictory fixture for press and release, connect and disconnect, or active and cleared state merely because the type accepts it.
Complete when: oracle provenance and fixture validity can reveal an authored test that encoded the defect as expected behavior.
3. Falsify the protection
In a reversible isolated change, remove the fix, invert the condition, or introduce the smallest representative fault. Run the narrow test first. Restore the source and rerun it green.
Do not accept a red caused by compilation failure, unrelated setup, or a different assertion.
Complete when: the intended assertion fails for the intended reason without the fix and passes after restoration.
4. Probe the model, not the example
For protocols, state machines, parsers, serializers, and event translation, run all applicable methods:
- mutation testing for changed or critical logic;
- property-based generation for invariants and equivalence classes;
- bounded fuzzing at structured untrusted-input boundaries;
- differential comparison against the independent oracle;
- deterministic seeds and replayable shrunk counterexamples.
For other changes, select methods from the contract. Scope campaigns to actionable code. Treat surviving mutants in critical logic as leads. Do not chase a universal mutation or coverage percentage.
Complete when: every modeled class is executed or excluded with evidence, generated failures are reproducible, and survivors are killed, justified, or recorded as gaps.
5. Exercise the real boundary and producer
Use integration tests for package, renderer, database, process, protocol, filesystem, or browser boundaries. Verify built, installed, bundled, serialized, or deployed artifacts when the claim depends on them.
When a browser, OS, SDK, or device produces the input, drive that producer at least once. Browser-generated events, real serialized bytes, process signals, or captured platform traces outrank hand-built objects. Synthetic fixtures may supplement the boundary drive only after their semantic validity is established.
Complete when: the test reaches the user-visible layer through the real producer, or the unavailable producer is named as a verification gap rather than replaced silently by a synthetic fixture.
6. Check determinism and cost
Repeat new tests, control clocks, randomness, ports, and external state, and record runtime. Put fast changed-code checks in the pull-request path and broader mutation or fuzz campaigns in scheduled runs.
Complete when: the test is reproducible, replayable, and placed at a sustainable cadence.
7. Report strength
Report the behavioral dimensions and exclusions, oracle provenance, fixture provenance, fix-absent red and restored green evidence, mutation or property results, real producer and boundary exercised, remaining survivors, and verification gaps.
Complete when: no claim of protection relies on a green suite, implementation-derived expectations, an implicit matrix, or semantically invalid synthetic input.
