| name | test-testing |
| description | Validate that an expensive integration/upgrade test actually catches the failure it claims to — by injecting a minimal, targeted fault and confirming the EXPECTED evidence appears in the logs, then reverting. Guards against tests that pass vacuously. Key idea most people miss: the injected failure is often RECOVERABLE (the system detects and excludes it and the test still PASSES), so the log evidence — not the exit code — is the real assertion. Use before trusting a green cross_binary / v118 / cluster run as proof of wire/state compatibility or malicious-party detection. |
Validate that a slow, opaque integration test (cross_binary,
v118_upgrade, cluster/simtest) actually catches the failure it claims to:
inject a minimal fault that should trip the property, confirm the
predicted log evidence appears, then revert. A green run alone proves
nothing — the catch is often recoverable, so the test passes either way and
the log line is the real assertion.
Target: $ARGUMENTS (the property/test to validate, e.g. "cross_binary state
continuity" or "honest validators reject a faulty one"). If empty, ask which
test and which property.
Full methodology — the two outcome shapes (hard-fail vs recoverable), the
7-step loop, which side to inject, the specificity control, and two worked
ika examples (cross_binary state continuity; cross-binary malicious-party
detection) — lives in dev-docs/playbooks/test-testing.md.
Read it and follow it; don't reinvent the methodology ad hoc.
The non-negotiables:
- The log is the assertion, not the exit code. Predict the exact log
string and which node's log it lands in before running; gate on grepping
for it at
$UPGRADE_TEST_DIR/validator-*/node.log.
- Pick the injection side deliberately — producer output vs a party's
outgoing round message are not interchangeable.
- Check specificity — only the faulty output/party should be rejected
while the rest stays healthy; blanket failure means you're measuring
collateral breakage, not the property.
- Revert every fault artifact — mark edits
// TEST-TESTING FAULT (temporary — DO NOT MERGE); afterwards git status is clean and
grep -r "TEST-TESTING FAULT" is empty.
