conformance-walk

name	conformance-walk
description	Conformance Walk — freeze-then-judge verification: coverage vs pass-rate, discrete verdicts, the adapter-registry interface, the 5 surface adapters (BE/FE/API/CLI/MCP). Use for any testable surface.

Conformance Walk (CW)

One-liner. ONE surface-neutral verification capability: enumerate every actionable unit of a surface → attach a FROZEN expectation → judge live/static state with a DETERMINISTIC oracle → report coverage HONESTLY (separate from pass-rate, machine-derived denominator) → RATCHET so a new unit without an expectation fails → TIER severity by surface-role. The core is built once; each surface supplies a small adapter. Users never choose "which method" — the matching adapter activates by the surface touched.

The four always-on MUST clauses are in rules/conformance-walk.md. This skill is the HOW: the methodology, the normative adapter-registry interface, the per-consumer adapter obligation, the 5-adapter table, the phase-action triggers, and the CW-vs-/redteam role split. Depth beyond the 80% here lives in three sub-files (loaded on demand): adapter-registry-interface.md, coverage-honesty-contract.md, reference-adapters.md.

1. The freeze-then-judge model (the heart)

• Freeze BEFORE observe. The expectation — the contract or transition a unit should hold — is pinned into a committed baseline BEFORE the unit is observed. An expectation back-fitted to whatever the code did is not an expectation.
• Floor ≠ expectation. "compiled / rendered / 200 / didn't crash" is the FLOOR. A different-than-expected effect is not a pass. (rule MUST-1.)
• Two expectation tiers. STRUCTURAL — machine-derived for every unit, zero hand-authoring cost, the mandatory floor of every unit. SEMANTIC — hand-authored or capture-then-human-freeze, only for the high-value units. Exact expected bytes (a toast string, an error message) are CAPTURED on first run and HUMAN-FROZEN before they assert — never invented from memory.

2. The expectation schema (the record every adapter emits)

One record per unit, four columns (surface-neutral):

• ACTUAL — what the surface exposes (signature/route/endpoint/flag/tool; observed state).
• INTENDED — the frozen expectation (contract / transition / status+schema+read-back).
• CONFORMANCE — the observed-vs-expected verdict + evidence.
• LINKAGE — edges to other units (a symbol exposed as an MCP tool surfaced in a FE route is ONE capability-record spanning symbol → endpoint → route; the integrated-linkage capture is the argument FOR unification).

3. Coverage vs pass-rate (the honesty contract)

• Denominator is machine-derived — enumerated from source/runtime, never hand-listed.
• Coverage (nothing unmeasured) is reported SEPARATELY from pass-rate (verdict outcomes). Two numbers, always. Coverage can be 100% while pass-rate is 80% — that is the honest state; the fails are the fix-list.
• No-vacuous-eval guard — a verdict counts toward coverage only if it carries ≥1 real assertion. Closes the assert(true) / expect(true) hole.
• No fabricated 100% by dropping non-pass rows or shrinking the denominator to the passing set.

Full machine-derived-denominator patterns per surface, the no-vacuous guard, the freshness+collision ratchet mechanics, and fail-closed integrity (empty-generation / mass-removal tripwires) are in coverage-honesty-contract.md.

4. The verdict taxonomy (discrete, closed)

Pass | Fail | Blocked | Retest | Skipped | Not-Run. Blocked (precondition down) ≠ Pass ≠ Fail. Retest (non-deterministic) ≠ Fail. Skipped (deliberately out of denominator). Not-Run (in denominator, never reached — a coverage gap, NOT a pass). (rule MUST-4.)

5. pass^k reliability

For a non-deterministic surface, a single Pass is weak evidence. Run the unit k times; report pass^k (k, passed, mean, std). A unit that passes 3/5 is Retest, not Pass. Reliability is a property of the verdict, not a separate gate.

6. The adapter-registry interface (NORMATIVE — design "A")

Each per-surface adapter supplies exactly three REQUIRED methods; the core owns everything else. This is the ratified freeze-line: the required interface is the INTERSECTION both shipped instances (SCM static-symbol, TOW route/interaction) already satisfy.

enumerate_units()            -> Iterable[Unit]     # machine-derived denominator; NEVER hand-listed
freeze_expectation(unit)     -> Expectation        # the frozen contract, pinned to the baseline BEFORE observation
oracle(unit, expectation)    -> Verdict            # deterministic; judges observed-vs-expected; from the discrete taxonomy

# Core-owned (surface-neutral): the {ACTUAL,INTENDED,CONFORMANCE,LINKAGE} record schema,
# coverage math (denominator + separate-from-pass-rate + no-vacuous guard), the freshness+
# collision ratchet over the committed baseline, the discrete verdict taxonomy, the two-tier
# oracle split (deterministic load-bearing / semantic advisory), the role-severity tier.

Capture-then-human-freeze is an OPTIONAL core-owned lifecycle (design "A"). The three methods above are the REQUIRED interface. For an adapter whose exact expected bytes are only knowable at runtime (a live surface: a toast string, an error message, a rendered post-state), the core ALSO offers an OPTIONAL capture-then-human-freeze lifecycle: on first run the semantic expectation is CAPTUREd, a human FREEZES it, and only then does it assert. A static adapter (SCM) whose expectations are statically knowable simply does not use it; a live adapter (TOW) does. Keeping it OPTIONAL and core-owned holds the required surface at three methods so a third adapter is never blocked by a two-instance-specific choice.

Adapter obligations the core validates: emits the core record schema; derives its coverage denominator machine-side; freezes before it observes; its deterministic oracle is the gate and any semantic output is advisory. Full method contracts, the Unit/Expectation/Verdict types, the capture-then-human-freeze lifecycle, and a worked adapter skeleton per surface are in adapter-registry-interface.md.

7. Per-consumer adapter obligation

For EACH surface your repo exposes, supply an adapter implementing the three methods, emitting the core record schema, deriving its coverage denominator machine-side. A BE-only repo supplies only the static-symbol adapter (no FE/browser cost). An integrated repo supplies several adapters and the CORE captures the LINKAGE across them. This is the TOW (#1137) "per-template instantiation" generalized to all five surfaces. The adapter is NOT a cascaded loom file — the skill specifies the obligation; each repo authors its own adapters for the surfaces it exposes.

8. The 5 adapters

Adapter	Surface	UNIT	Oracle (freeze → judge)
static-symbol (BE) = SCM	SDK / library	public fn / struct / enum	signature + spec-conformance + test-tier + AST-tamper, judged WITHOUT running
route/interaction (FE) = TOW (#1137)	FE / UX	route + interactive element	endpoint-fired + DOM/a11y read-back + record read-back, judged AFTER freezing the expected transition
endpoint (API)	HTTP / REST	endpoint	status + response-schema + persisted-effect read-back
flag (CLI)	CLI	subcommand / flag	exit code + output-shape + state-effect read-back
tool (MCP)	MCP	tool call	result-schema + side-effect read-back

Auto-activation by touched surface; the reference adapters are SCM (BE) and TOW (FE) — the other three are thin unit-enumerator + oracle over the proven core. The two reference implementations, distilled to the interface as the worked examples a new adapter author copies, are in reference-adapters.md.

9. Phase-action triggers

CW is a STANDING capability (record + CI gate always-on) with phase-specific ACTIONS. Freeze-then-judge requires the expectation to precede the observation, so declaration is upstream:

• analyze: enumerator runs → surface INVENTORY + baseline coverage (the un-expectationed frontier).
• todos: each todo adding a unit DECLARES its frozen expectation — the freeze, BEFORE code.
• implement: records populate incrementally; the FRESHNESS GATE fires — no new unit without a declared expectation.
• redteam: THE PRIMARY GATE — judge every unit, structural-BLOCK failures, hand the human the semantic-ADVISORY worklist.
• codify: absorb this round's findings into the role-severity tier prior (self-recalibration); record the frontier debt.
• deploy (+ every CI run): the freshness gate + collision ratchet stand as the structural merge gate — no ship with a coverage regression or a new-unit-without-expectation.

One line: freeze @ todos → generate + freshness-gate @ implement → PRIMARY adversarial walk @ redteam → absorb + recalibrate @ codify → ratchet merge-gate @ deploy/CI.

10. CW vs /redteam (role split — not a replacement)

CW is the STANDING, mechanical, PRE-/redteam gate: it front-loads the deterministic half (structural BLOCK on irrefutable facts) and hands the human a PRE-COMPUTED per-unit semantic-ADVISORY worklist — the exact unit + the exact question — instead of making the human rediscover it. /redteam remains the ADVERSARIAL human/agent judgment layer that adjudicates the advisory worklist and hunts for what no oracle can enumerate. CW makes /redteam's budget go to ADJUDICATION, not discovery; it does NOT replace the adversarial round.

11. The role-severity tier (self-recalibrating)

Severity = f(verdict-status, surface-role), NEVER unit count. The role/value prior is evidence-driven and self-recalibrating (a defect-history-hot surface promotes ITSELF toward a blocking tier; a value-ranked journey carries a user-anchored value-anchor) — never a frozen hand-ranking, never a hardcoded denylist. Only irrefutable structural facts on a role-relevant unit BLOCK; semantic statuses are the advisory worklist.