verifier-completeness

name	verifier-completeness
description	Use when writing, auditing, or trusting a checker that decides whether an answer is correct — a grader, a reward function, an equivalence test, an assertion-based test oracle, a judge. The reflex is to test SOUNDNESS (a wrong answer must never be accepted). This skill forces the other direction, which a soundness test structurally cannot see: COMPLETENESS — is every CORRECT answer accepted? Fires on "is this verifier right", "the pass rate looks too low", "the pass rate is identical across every variant", "compare two answers for equality", "check symbolic/numeric equivalence", or before trusting any score a checker produced.
metadata	{"short-description":"Audit a checker for FALSE-NEGATIVES (correct answers rejected), not just false accepts; and decide polynomial equality exactly before sampling"}

verifier-completeness

A verifier has two directions of error:

• Unsound — accepts a wrong answer. Everyone tests this.
• Incomplete — rejects a correct answer. Almost nobody tests this, and a soundness test can never find it, because a correct-answer rejection mints no false accept.

Incompleteness is the more expensive failure whenever the verifier's output feeds back into behavior. As an eval it under-reports capability: a checker that rejects a valid alternative notation reads as "the task is hard" when the task is actually at ceiling. As a training or selection signal it is worse — every correct attempt that gets scored wrong actively pushes the producer away from correct behavior. Re-scoring afterwards fixes the numbers; it does not undo anything that was already learned from the bad scores.

A worked example of the shape: a symbolic-math checker with no LaTeX handling parsed a bare e as a free symbol, so -2x \sin(x^2) (against gold -2*x*sin(x**2)), \frac{1}{2} e^{2x} (against exp(2*x)/2), and e^x were all scored wrong. Nothing in the soundness suite could see it.

When to use this skill

• You wrote, or are about to rely on, a verifier — symbolic, execution-based, string-matching, or judge-based.
• A pass rate looks suspiciously low, or suspiciously uniform across variants that should differ. A verifier ceiling reads exactly like "no signal here".
• A change made a system worse on a metric that verifier produced.
• The verifier's parser or normalizer was touched. "It parses" is not "it parses correctly".
• A checker decides equality by evaluating both sides at a few sample points.
• You are writing an equivalence test where the same value has several valid renderings — LaTeX vs source syntax, 1,000 vs 1000, differing whitespace, ordering, or units.
• Not for: judging whether an individual answer is right. This is about the checker, not the answer.
• Not for: deciding whether a task was completed correctly, and not for any workspace action — it audits accept/reject logic, not a work product and not a live service.

Procedure

1. Pick canonical golds that exercise the notation classes the domain actually produces. For symbolic math that is: plain source syntax, LaTeX (\sin, \frac{a}{b}, e^{x}), unicode and caret superscripts, prose-style implicit multiplication (2x), fraction vs. division, and exp(x) vs e^x. Six golds × at least five renderings = 36 pairs is a usable floor. For a text answer the classes are whitespace, case, punctuation, unit spelling, and number formatting (1,000 / 1000 / 1e3). For code, the classes are output formatting and ordering.

2. Assert through EVERY entry point. A verifier with two doors — say a scoring function and a separate eval oracle — can be complete at one door and incomplete at the other, even when both sit on a shared parser. Test each door by name.

3. Keep the battery sound-aware. Every gold and every rendering in a case must be genuinely equal, so "accept" is always the correct verdict. This battery can only ever detect an under-accept; it must never be able to license an over-accept. Do not put wrong answers in it — that direction belongs to the soundness tests.

4. Wire it in as a test, fail-closed. If the verifier's dependency is missing, the run is not verifiable: skip loudly, never pass silently.

5. Replace any sampling-based equality test with the exact-first ordering in the sub-section below, if the verifier has one.

6. Report completeness as its own number, following the honesty rules.

Deciding equality: exact before sampling

If the verifier falls back to evaluating both sides at a few sample points to decide equality, a FIXED sample set is a soundness hole that can be collided with by construction: any wrong answer whose difference from the gold is a polynomial with roots exactly at the sample points reads as "equal". Against a checker that sampled the four fixed points {0.5, 1.5, 2.5, -1.3}, the answer

sin(x) + (x - 0.5)(x - 1.5)(x - 2.5)(x + 1.3)

is accepted as equal to sin(x).

The fix is ordering, and it is cheap:

1. Form the residue candidate - gold and simplify it.
2. If the residue is a polynomial, decide it EXACTLY — sympy.Poly(residue, x).is_zero. A nonzero polynomial of degree d has at most d roots, so no finite point sample can ever prove it zero, but is_zero decides it outright. This step alone kills the whole collision class.
3. Only a genuinely transcendental residue should reach a numeric probe, and that probe should be a fixed-seed battery of many points (at least 32) over a wide domain, requiring agreement at every point where both sides are defined. Fixed seed means byte-reproducible: no wall-clock, no fresh entropy, so the verdict cannot change between runs.

This ordering is one-directional and safe: a truly equal pair has a residue that is zero everywhere, so adding the exact check introduces zero new false rejects.

Honesty rules

• A clean audit is a valid result. If the battery finds nothing, say so plainly. Do not invent a notation class to manufacture a finding.
• Re-scoring fixes numbers, not artifacts. Anything already produced or trained under the broken verifier stays suspect until it is redone. Keep those two statements separate; do not let a corrected score be read as "everything downstream is fine now".
• Report the completeness pass rate as its own number, next to the soundness one. "The verifier is correct" is two claims, and they fail independently.

Worked example

Input: a symbolic-math grader reports a base pass rate of 0.52 that barely moves across variants that should differ, and its normalizer was edited recently.

Approach: both the flat-across-variants signature and the touched-normalizer signature point at the checker, not the answers. Build a fixed battery: six golds — a derivative, an integral, an exponential, a trig expression, a fraction, and a radical — each rendered six ways (plain source -2*x*sin(x**2), LaTeX -2x \sin(x^2), \frac{1}{2} e^{2x}, a caret superscript, a unicode superscript, implicit multiplication 2x). Every pair is genuinely equal, so every expected verdict is accept. Run all 36 pairs through the scoring function and through the eval oracle by name, since one door can be complete while the other is not. A bare e parsed as a free symbol shows up immediately as a whole rejected column. While in there, check whether equality falls back to a fixed point sample — if so, put sympy.Poly(residue, x).is_zero in front of it. Then report the completeness pass rate as a number of its own, and state that anything already scored under the old checker stays suspect until re-run.

References

None. The battery, the entry-point rule, and the exact-before-sampling ordering are all constructed from the verifier under audit, so no bundled file is required.