verify-completion-signal

name

verify-completion-signal

description

Use after any long-running subprocess, eval driver, CI gate, or job pipeline run reports apparent success (exit 0 / phase=DONE / "all tests passed") — before accepting the result. Catches the silent-success trap: a process can exit 0 without completing its intended work, and inference from "no error" to "success" is wrong. Require an explicit positive completion artifact (an output file, pass-count line, verdict.json with a status field, non-empty results directory).

Verify a Completion Signal

Overview

Exit 0 means "the process finished without crashing." It does not mean "the outcome you wanted exists in the world." The gap between those two statements is where silent failures live — and they look identical to successes until you check the artifact directly.

Core principle: require a POSITIVE completion signal. An absence of error is not a positive signal. A positive signal is a concrete artifact that can only exist if the work was done: a file with non-zero size, a X passed line in stdout, a verdict.json with a populated status field, a non-empty results directory.

This is a cross-cutting reflex. It applies to any tool, subprocess, CI gate, or eval driver that can exit cleanly without completing its work.

When to use

• A long-running job pipeline reaches phase=DONE or exit=0 — before accepting the result, check the output directory contains actual result files.
• A pytest product-gate appears to exit 0 with no X passed line in stdout (faulthandler/pluggy crash path: the interpreter exits before the test runner prints its summary).
• A driver prints === DONE === but the results directory is empty or contains only the log.
• Any subprocess piped through tee, || true, or a shell wrapper — the last command's exit code shadows the real one.
• An agent or eval loop self-reports "complete" or "success" with no artifact cited.

NOT for: validating that the outcome is correct once the artifact exists — use the task's own metric after the completion signal is confirmed.

The method

Step 1 — Name the artifact. Before the run starts, state what must exist on disk when it succeeds. Not "the script ran." The real thing: output_dir/master.out + at least one .json in output_dir/, a X passed line in captured stdout, a non-empty results/<run_id>/ directory.

Step 2 — After apparent success, check the artifact independently. Do not re-read the tool's own return value or log line. Go to the filesystem / stdout buffer directly:

• ls -lh output_dir/ — does it contain files with non-zero size?
• Grep stdout for passed (not just exit code): echo "$OUTPUT" | grep -E '[0-9]+ passed'
• Read verdict.json and confirm status is populated, not a placeholder like STEP1_INCOMPLETE.

Step 3 — Treat a missing artifact as a failure, not as "probably fine." If the artifact is absent, the run did not complete — even if exit=0, even if phase=DONE. A DONE marker with an empty output dir is a bug, not a success.

Step 4 — Identify the silent-success class and fix it upstream. Three common classes:

Class	Symptom	Root cause	Fix
DONE-with-empty-output	phase=DONE, exit=0, output dir has no result files	the pipeline's DONE gate does not verify n_files > 0 in the output dir	Until fixed: ls output_dir/ manually after every run; count non-log files
Gate-exit-0-but-crashed	pytest exits 0, stdout has no X passed line	native faulthandler / pluggy crash (_pytest.faulthandler path) exits the interpreter before the test summary prints	Require the X passed pattern in captured stdout; a zero-line output is not a pass
Pipe-swallowed exit	cmd | tee log.txt exits 0 regardless of cmd's code	bash returns the exit of tee, which always succeeds	set -o pipefail in every shell driver; capture ${PIPESTATUS[0]} explicitly

Common mistakes

"Exit 0 = success." Exit 0 = "the process terminated without an unhandled signal or explicit failure exit." A script that finds nothing to do, a process that crashes via _exit() before cleanup, and a subprocess wrapped in || true all exit 0. None of them completed the work.

"The DONE marker was printed." The DONE marker in master.out (=== DONE === run=R exit=N) means the driver reached its terminal block and printed. It does not mean the result-transfer step succeeded or that the output directory has result files. A verdict-classifier function that checks the content of transferred results — but only if files exist — can silently return None on an empty output dir, masking the gap by falling through to a generic DONE outcome.

"All tests passed" (from the agent's summary, not stdout). An LLM summarizer reading test output can report success from a partial log — if the stdout was truncated or the crash happened after the test collector ran but before the reporter printed. The X passed line must appear in the raw captured stdout, not in a summary inferred from it.

"The transfer verified it." A results-transfer verification step can check only the files listed in an expected_files list. If that list only names a log file and the actual results JSON was never written, the check verifies the log and declares success. The check is only as complete as the list you pass in.

Field grounding

• arXiv:2509.14347 (Illusion of Success, 2025): 11% of industrial CI jobs are rerun; "silent failures" — exit 0 but the intended artifact is missing — are the top-3 cause: artifact operation errors 28%, caching errors 23%, ignored non-zero exit codes 18%.
• The | tee class is the canonical CI trap (three weeks of broken builds shipped green: bash returns tee's exit, not the compiler's; the fix is set -o pipefail).
• pytest faulthandler path: when a C extension or pluggy hook crashes the interpreter before the test-session end hook runs, _pytest.faulthandler may write a traceback to stderr but the Python process exits via _exit(status) — which skips atexit, sys.exit, and the test reporter. The exit code may be 0 if the crash happened in teardown rather than in a test itself.

Receipts

• A long-running job runner reached phase=DONE exit=0 with zero result files transferred back to the output directory. The verdict-classifier silently returned None on an empty output dir, so the final outcome fell through to a generic DONE. No n_files > 0 guard existed yet.
• A full product-gate test suite faulted via a native pluggy crash — exited 0 with no X passed line in stdout. The gate appeared green; it delivered no result.
• Both were caught by the same reflex: "the run says DONE / green — where is the artifact?"

Companion skill: verify-the-claim-not-the-label.

The positive-label corollary (see verify-the-claim-not-the-label)

This skill verifies COMPLETION artifacts after a run (exit codes, DONE markers, output files). The same reflex extends to any CACHED or ASSERTED positive-state LABEL — 'this is resolved', 'this is passing', 'the cache is ready', 'the cause is X', including labels YOU just applied. That generalization lives in the canonical skill verify-the-claim-not-the-label. Rule: before stating a bug is 'resolved' / a test is 'passing' / a mechanism is 'X', confirm the positive artifact (the 'N passed' line, the per-sample distribution, the file size > 0) — not the label. A label is a hypothesis.