| name | audit-the-oracle-coverage |
| description | The dangerous case is not "no oracle" — it is a WEAK oracle you falsely believe is exact. Before trusting any test suite, metric, benchmark, eval, or proof as if it settled the question, state explicitly what it CERTIFIES and what it does NOT — the gap between the proxy and the target. The failure it prevents: false certainty from a passing check that silently misses the spec (tests green but the spec was wrong), proxies value (offline metric up, user value flat), or has drifted (backtest on a system that has since changed). Triggers whenever a check comes back positive and you are tempted to call the thing "verified", and as the mandatory middle step between build-the-oracle-before-the-answer and triangulate-without-oracle. |
audit-the-oracle-coverage
The matched pair build-the-oracle-before-the-answer (oracle exists) vs
triangulate-without-oracle (no oracle) is a false binary. Most real work
lives in the middle: a partial or proxy oracle that certifies something
but not the whole claim, and the characteristic failure is treating that
partial certificate as a full one. This skill is the three-way partition the
binary was missing — exact / partial-proxy / none — and it routes each part
to the right discipline instead of laundering a weak signal into confidence.
Fuses build-the-oracle-before-the-answer (P-1) with prove-the-coverage-denominator
(P-11) applied to the oracle itself, and routes the uncovered residual to
triangulate-without-oracle (P-9). Organon: P3 fault-model + P11 + P19 oracle +
P37 residue-seed. Phase: JUDGE.
Procedure
-
State what the oracle actually certifies. Write the precise claim the
check establishes when it passes — not the claim you wish it established.
"All 412 tests pass" certifies the behaviors those tests assert, not "the
code is correct."
-
Name the gap between proxy and target (P3). What is true of the target
that the oracle does not see? Spec ambiguity it never probed, inputs it
never ran, the distribution it was built on but no longer matches, the
value it proxies but is not. If you cannot name a single thing the oracle
misses, the audit did not happen — that itself is the red flag.
-
Partition the claim (P11 over the oracle). Label every part of what you
are asserting as:
- covered — the oracle directly certifies it;
- proxied — the oracle correlates with it but with a known, named gap;
- uncovered — the oracle says nothing about it.
-
Route each part to its discipline. Covered → trust it. Proxied →
either strengthen the oracle (build-the-oracle-before-the-answer on the
gap) or carry the gap as explicit risk. Uncovered → if it is testable,
build an oracle for it; if it is irreducibly normative/strategic, route to
triangulate-without-oracle and name the owner.
-
Report the partition, not a binary verdict. "Verified" is replaced by
"covered: X; proxied with gap: Y; uncovered, routed to Z."
The gate
You may not report a claim as "verified" until its parts are partitioned into
covered / proxied / uncovered, with at least one named thing the oracle does
not certify. An audit that finds the oracle covers everything is presumed to
have not looked — partial oracles always miss something; name it or the gate
fails. The one escape: an oracle that provably enumerates the claim's entire
finite case space may be declared exact, and then the finiteness argument
itself is what the audit must produce. The proxied and uncovered parts must
be routed, not folded into the covered set.
Example
A migration's test suite passes, and the instinct is "settled, ship it." The
audit: covered = the 412 asserted behaviors; proxied = "preserves user
settings" (the tests check three settings shapes, correlated but not the full
space); uncovered = users with null locale and deleted workspace memberships
(no test exists). Routing the uncovered case to a targeted fixture (build an
oracle for the gap) finds one settings row that silently drops — exactly the
case "all tests pass" was about to certify away. The binary "verified vs not"
would have shipped it; the coverage partition caught it.
