| name | verify-the-claim-not-the-label |
| description | Use when a status label (DONE / RESOLVED / PASSING / CACHED / READY / SUCCESS / "it completes") is the only evidence that something works. Triggers on: "the gate is green", "that bug is resolved", "the cache is ready", "I ran it and it passed", "the test completed", "a generated report exists but is empty", "CI shard 1 passed but shards 2-3 never ran", "a cache said READY but was corrupt". The discipline: a SUCCESS label is a HYPOTHESIS until the bytes / per-sample data / real code confirm it. Extends SILENT-SUCCESS REFLEX (already in CLAUDE.md) to the agent's OWN claims and OWN live checks, not just external signals. |
Verify the claim, not the label
The rule: a SUCCESS / DONE / RESOLVED / PASSING / CACHED / READY label is a
HYPOTHESIS, not ground truth. Verify it against the bytes, the per-sample data,
or the real code before trusting it — including claims you just made yourself.
This extends the SILENT-SUCCESS REFLEX (CLAUDE.md, 2026-06-25 arc) to the hardest
case: catching your OWN "it works" conclusions before they propagate.
Why labels mislead
A label encodes what was believed at the moment it was written, not what the system
does. Tests pass against what the author expected, not the full behavior space. Docs
drift the moment code changes. A green CI check can prove nothing about the path that
actually fails in production (siliconopera.com, 2026). The longer a label has been
trusted, the more dangerous: "a test passing for 18 months on a stale assumption is
confidence you never paid for" (dev.to/sophielane). The trap deepens when the same
agent writes implementation and verification — both encode the same misunderstanding
(plumblinehq.ai, beingagent.org).
Red-flag table
| Label you see | What it actually guarantees | What to check instead |
|---|
"exit 0" / "DONE" | The process did not crash | Was a real artifact produced? (count files, check size > 0) |
"tests passed" | The tests that exist passed | Do those tests exercise the code path that could fail? |
"RESOLVED / FIXED" | Someone believed it was fixed at commit time | Run the failing scenario against the actual fix in the actual env |
"CACHED / READY" | A cache entry exists | Is the cached file non-zero and uncorrupted? Load it |
"extraction floor (must be harness bug)" | A narrative that sounds mechanistically complete | Pull per_item data; distinguish real floor from under-elicitation |
"it completes" | The happy path ran | Did ALL the sub-tasks complete? (chunked gate, multi-bench suite) |
| Comments / docs / CLAUDE.md lore | State of the system at time of writing | grep the ACTUAL running code; lore has a shelf life |
Verification protocol (3 questions)
- What concrete artifact proves this is done? (non-zero file, specific stdout line,
per-item score distribution — not just an exit code)
- Would I have seen a different label if it were broken? If not, the label is
decorative (plumblinehq.ai: "a green is only evidence if a broken system would have
gone red").
- Am I reading the code that runs, or the doc that describes what was intended to
run? Code is ground truth; comments rot (beingagent.org).
Illustrative cases — where this discipline pays off
A "resolved" bug that was actually a 3-layer wrong-lore chain. A CI gate was
believed "BROKEN" (exit crash). Layer 1 diagnosis blamed one dependency. Layer 2
blamed another. Both wrong — the real cause was a cumulative resource issue that only
triggers when the full file set runs together. None of the three layers was verified
against the running code before being recorded; each was accepted on narrative fit
alone. Lesson: a plausible mechanism story is not a verified mechanism. Grep the
real crash stack; test the hypothesis before banking it as lore.
A cache said READY but was corrupt. An artifact existed on disk with a "cached"
marker file next to it. Assumed: usable. Reality: the write had been interrupted and
the artifact was truncated to zero bytes — the marker was written, the payload wasn't.
Lesson: presence of a cache entry is not integrity. Check file size and actually
load the thing before trusting the marker.
A generated report exists but is empty. A report-generation step exited 0 and
produced a file at the expected path. First read: "the report generated successfully."
Opening the file showed zero rows — the generator had silently no-op'd on an empty
input set upstream. Lesson: "the file exists" and "the file has content" are
different claims; check both.
CI shard 1 passed but shards 2-3 never ran. After shard 1 of a 3-shard CI run
finished green, the claim "CI passed" was written before shards 2 and 3 had even been
dispatched. The full suite was required to establish the real outcome. Lesson: a
partial run completing is not "it completes." Count the sub-tasks; confirm the final
artifact against the expected total.
Applying it to your OWN claims
The CLAUDE.md SILENT-SUCCESS REFLEX originally targeted external signals (DONE-with-empty-
harvest, gate-exit-0-but-crashed). This skill extends it inward: your own live check is
also a hypothesis. Before writing "I ran it and it worked," confirm the output file is
non-zero, stdout contains the expected terminal line, and per-sample data matches the
interpretation. Trust comes from the evidence, not the label.
Cross-references
- CLAUDE.md "SILENT-SUCCESS REFLEX" — the parent discipline
