| name | runtime-parity-verification |
| description | Use in Phase 4/5 (Validate/Iterate) for ANY change to a user-visible flow — web, macOS, iOS, agent, or CLI/TUI — before claiming "done". Verifies the RUNNING app's core flow by cross-checking the rendered/queryable UI against the backing source-of-truth (DB/API/daemon/tool-state), screen-independently. Triggers — "verify it works", "confirm the fix", "does X actually work", "nothing happens when I…", "not showing", "shows empty/wrong data", uiTarget != null, or any "action does nothing / data not displaying / stale projection" symptom. Compile-green and a passing unit test do NOT satisfy this; neither does a screenshot alone. |
| user-invocable | false |
Runtime Parity Verification
The recurring failure this prevents: an agent ships UI/feature work, reports "compiles green / tests pass / committed," and never confirms the running app's core flow actually works — or confirms it only by screenshot, which is screen-dependent and gets silently deferred (e.g. the user's display locks). The whole "X doesn't work / isn't showing / shows mock/empty despite real data / stale projection" bug class is a divergence between what the UI renders and the authoritative backend state. One cheap, headless check catches the entire family.
The invariant to assert
For the user's core flow, the rendered/queryable UI state must equal the authoritative backend state.
- If the backend holds N items and the UI shows 0 / "empty" → FAIL (projection diverged).
- If an action is supposed to create/change state, assert both the backend delta and the UI reflecting it.
This is stronger than "did it render" (which UI validators check). It is also stronger than "build + tests pass" (which never exercises the running flow). It must be screen-independent so it cannot be skipped when no display is available.
Platform recipes — source-of-truth (left) ⇄ UI probe (right)
| Platform | Authoritative source-of-truth | Screen-independent UI probe |
|---|
| Web | API response JSON / DB row count | Host browser or ui-validator / IBR scan; assert via data-testid selectors |
| macOS (AppKit/SwiftUI) | backend store / daemon via CLI or socket query | native-ax-driver Accessibility tree (cursor-free). Reference impl: easy-terminal tools/smoke_launch.py |
| iOS / watchOS | backend / store | simulator AX via idb ui describe-all or XCUITest queries |
| Agent / LLM app | tool-result or persisted state | the agent's returned/rendered output — assert the claim matches the actual tool/state, not just that text was produced |
| CLI / TUI | process / file / db state | captured stdout / TUI buffer |
Procedure
- Identify the core flow changed (the thing a user does: launch a thing, submit a form, see a list, run an action).
- Capture source-of-truth before and (if the flow mutates) after the action — a query that does NOT go through the UI layer.
- Probe the UI headlessly with the platform driver and extract the rendered state (counts, presence/absence of an empty state, the new item).
- Assert parity: rendered == source-of-truth. On a mutating flow, assert the backend delta AND the UI reflecting it.
- Encode it as a per-repo smoke (a script that returns non-zero on divergence). Validate the smoke is real: confirm it returns non-zero on a known-broken state, not just zero on green — a check that cannot fail is worthless.
- Gate on it: run the smoke before any "done" claim. Never substitute compile-green, a passing unit test, or a screenshot.
Doc ↔ interface parity (documented CLIs, tools & flows)
A second parity gap, same shape: the documented interface diverges from the
runtime interface, and the CLI silently accepts malformed input. A green
unit-test suite over the core logic structurally cannot catch either — tests
exercise the functions, not the CLI surface or the commands the docs tell a user
to run. A tool/CLI/flow is not "done" until:
- Every command in a flow doc runs against the live CLI. Extract each
documented command (fenced shell blocks in the flow's
.md) and validate its
subcommand + every --flag against the real interface — --help for a
Python/argparse CLI (side-effect-free), the parsed-flags list in source for a
node CLI. A documented flag/subcommand that doesn't exist is a FAIL. (This is
the "doc says --params, the CLI never had it" defect.)
- Every CLI boundary rejects malformed input. Feed each command a typo'd
sub-argument / unknown enum value and assert a nonzero exit with an error —
never a deadlock, never a silent default substitution. (A typo'd argument that
auto-creates a junk record and hangs the loop, or a bad value silently swapped
for a default, both ship a tool the user can't trust.)
Validate the check the same way step 5 demands — mutate the real files (a
bogus doc flag; a disabled validator) and confirm the check FAILS; a check that
can't fail is worthless. A subtle trap seen in practice: input-validation cases
that run against an absent state file pass even with validation disabled,
because the missing-file crash also exits nonzero — the oracle confounds
"rejected" with "crashed." Run the malformed-input cases against a real, valid
state so a reject is the ONLY cause of a nonzero exit.
Reusable exemplar: groundwork designer/conformance/flow_cli_check.py — scans
references/*.md + SKILL.md, validates every documented python3 -m … / node … command against the live CLI, feeds each a malformed value, and
--selftest-mutation-proves it bites. Wired to fire via scripts/check.sh →
npm test + a committed .githooks/pre-push (a check that only runs when recalled
is dormant — put it in the gate).
Anti-patterns (each one shipped a real bug)
- "Build is green, committed — done." → compile ≠ runtime; never exercised the flow.
- "Here's a screenshot, looks right." → screen-dependent; deferred when the screen locks; can't diff against truth.
- "The UI rendered something." → rendering ≠ correct data; an empty state renders fine while the backend has 100 rows.
- Verifying only the backend (CLI/API works) without the UI, or only the UI without the backing truth — the bug lives in the gap between them.
Build-loop integration
- Phase 4 Review sub-step B / Phase 5 Iterate: when
uiTarget != null OR the diff touches a user-visible flow, a runtime parity check is required. The existing drivers do the probing — web: ui-validator; macOS: native-ax-driver / IBR scan_macos; iOS: idb. THIS skill adds the missing step: cross-check the probe against source-of-truth, and keep a validated per-repo smoke.
- Phase 4 sub-step G (
verification-before-completion): for app/UI changes, "confirm output" includes the runtime parity smoke, not only test/build/lint. For a CLI / tool / documented-flow change, it also includes the doc↔interface parity smoke (every documented command runs; every boundary rejects malformed input), likewise validated by mutation.
- The
verify skill ("run the app and observe behavior") is the manual counterpart; this skill is the automatable, source-of-truth-anchored form.
Origin lesson: build-loop-memory lessons/2026-06-08-pattern-runtime-ui-source-of-truth-parity-verification.md (easy-terminal launch/no-pane bug — UI projection diverged from daemon, missed across a whole UI pass because verification was compile-green + screenshot-only).
