ai-dlc-verify

name

ai-dlc-verify

description

Browser verification for AI-DLC — turns the G4 demo script into reproducible, screenshotted evidence across Local, Staging and Prod. Drives a real login, walks a feature's verification steps at each declared viewport, captures desktop and mobile screenshots, and generates a markdown evidence report plus a short PR draft. Stack-agnostic — everything project-specific lives in the `verify:` block of `.ai/aidlc.yaml`. Degrades silently: a project with no verification config, or with auth but no credentials, ends without screenshots and without blocking any gate. Use whenever the user says "verify", "screenshot", "evidence", "responsive check", "PR draft", or is closing a Unit of Work at G4. The workflow, the gates and the controller come from ai-dlc-core.

AI-DLC verify

Stock AI-DLC closes a Unit of Work by having a human run its Demo script in front of another human. That is a real gate and it is deliberately manual. This package keeps the demo and adds a machine-reproducible record of it: the same script, driven in a browser, against each environment, at each viewport, with screenshots that end up in the pull request.

The workflow, the gates and the controller all come from ai-dlc-core — read its SKILL.md first and run aidlc status before anything else. This package supplies only the verification half of G4.

The ladder is the whole design

A verification tool installed across many repos will meet projects that have no login, no staging environment, or no credentials on this machine. It must be uneventful in all three cases. Verification resolves at runtime to exactly one rung, and only the third can block a gate:

Rung	Condition	Behaviour	Effect on G4
not applicable	no verify: block in .ai/aidlc.yaml	nothing runs; no evidence checkboxes are written into uow.md at all	none
skipped	configured, auth required, credentials absent or blank	one line of output, no browser launched, no screenshots, exit 0	none
capable	configured, credentials present	full run; evidence required	blocks until green

python scripts/verify.py .ai/features/<slug> --doctor

--doctor is the first thing to run, and the answer to "should I write verification checkboxes into this UoW". Run it at G3, before the UoW template is written — because the checkboxes are the enforcement, and writing them into a project that can never satisfy them turns G4 into a wall.

Skipped is not failure. It exits 0, writes nothing, and says why in one sentence. A project with auth and no credentials in env just ends.

Commands

verify.py <feature-dir> --doctor              # which rung; changes nothing
verify.py <feature-dir>                       # run; screenshots to evidence/
verify.py <feature-dir> --write               # run, then generate 08-evidence.md
verify.py <feature-dir> --env local           # restrict to one environment
verify.py <feature-dir> --manual-login        # headed; you log in, the session is saved
verify.py --version                           # aidlc_verify 0.1.0 (ruleset 4)
evidence_check.py <feature-dir>               # validate the claims; exit 1 on a lie

Exit codes match uow_graph.py: 0 ok · 1 failed · 2 usage error.

Everything is Python. verify.py and evidence_check.py are stdlib only, like every script core ships — a machine that has never installed a browser driver can still resolve the ladder, read run.json and validate evidence someone else produced. scripts/runner/run.py is the one file with a dependency, and it runs as a separate process so that stays true:

pip install -r scripts/runner/requirements.txt     # playwright
playwright install chromium                        # the browser itself

If your system Python is externally managed — Homebrew, Debian — put Playwright in a venv and point the runner at it; nothing else changes:

export AIDLC_VERIFY_PYTHON=~/.venvs/aidlc-verify/bin/python

--doctor reports the runner's absence as a separate line from the rung, because a missing browser is a machine problem and a missing credential is a project one.

Output layout

Continues core's numbering rather than opening a parallel tree:

.ai/features/<slug>/
├── 00-intent.md … 06-traceability.md      ← core
├── 07-verification.md                     ← hand-written: the feature's verification steps
├── 08-evidence.md                         ← generated by --write
└── evidence/
    ├── run.json                           ← generated: verdicts, sha, browser, timing
    ├── <env>/<viewport>/<step-id>.png      ← generated
    └── contact-sheet-<env>.png            ← generated: one composite per environment

08-evidence.md and evidence/ are generated. Add them to the repo's .gitignore beside core's three generated entries, along with the two paths that hold secrets:

.ai/features/*/08-evidence.md
.ai/features/*/evidence/
.ai/credentials.env
.ai/.auth/

.ai/credentials.env is not covered by a .env.* pattern — that matches a basename beginning with .env, not one ending in it. Check with git check-ignore -v rather than assuming.

Configuration

One block in .ai/aidlc.yaml, beside the keys core already reads. No second config tree.

verify:
  environments:
    local:   { url: http://localhost:3000, required: true }
    staging: { url: "${STAGING_URL}",      required: true }
    prod:    { url: "${PROD_URL}",         enabled: false, writes: false }
  viewports:
    desktop: { width: 1440, height: 900 }
    mobile:  { width: 390,  height: 844, isMobile: true }
  auth:
    recipe: clerk-hosted        # clerk-hosted | form | storage-state | none
    landing: /
    ready_when: { gone: "text=Loading" }
  warmup:
    attempts: 1                 # one unjudged request per env, before anything is verified
  failure_signals:
    - { selector: ".Toastify__toast--error", message: "app surfaced an error toast" }
    - console_errors

auth: can also be set per environment (environments.<name>.auth), which is how one system with two frontends signs in to each of them. See references/config-schema.md.

${VAR} interpolates from .ai/credentials.env, which holds every secret and every URL you do not want in a config file. That file must never be committed — core's sync.md guidance applies, and the safest posture is ignoring all of .ai/.

An environment with enabled: false is configured but never run. That is how you keep production in the model — documented, with writes: false — without ever pointing a browser at it. Only required: true environments gate G4.

viewports is arbitrary and project-chosen. Pick widths that straddle the breakpoint your app actually branches on, and avoid the breakpoint value itself: at exactly the cutoff, CSS media queries and a JS matchMedia hook can disagree about which side you are on.

Read references/config-schema.md for every key, references/login-recipes.md for the auth presets, and references/templates.md for the two artifact shapes.

Definition of done

Verification is not "a screenshot exists". A step passes only when the page reached the declared ready state, no configured failure_signals fired, no console error was logged, and every claim in its Assert column held. A screenshot of a blank page that silently 403'd is worse than no screenshot, because it looks like evidence.

The Assert column is what separates "the page loaded" from "the number is right", and a step without one asserts nothing at all. Skip it and a run can be entirely green over a feature that computes the wrong total — the most expensive failure this package can produce, because the green will be believed. Assertions also make a deliberately red step the honest way to record a known defect: state the correct behaviour, let the broken build fail it, and the screenshot becomes the bug report. See references/templates.md.

When --doctor reports capable, uow.md gets the block from references/templates.md:

## Verification evidence
- [ ] `verify.py <feature-dir> --write` green on every required environment
- [ ] Evidence exists for every AC in `verifies`, at every declared viewport
- [ ] `08-evidence.md` regenerated and its commit sha matches HEAD
- [ ] PR draft copied and contact sheets attached to the PR description

Core refuses pass G4 while any - [ ] in uow.md is unticked, and it scans the whole file — so this section is enforced exactly like the definition-of-done. That is the only lever core offers, and it is the reason the block must not be written on the other two rungs.

Ticking a box is still a claim. evidence_check.py is what turns it into a fact: it reads run.json and the UoW frontmatter and confirms that every AC in verifies: has a passing screenshot at every required environment × declared viewport, that no step failed, and that the recorded commit sha matches HEAD. A skipped run reports N/A, never a failure.

Reference files

Read at the phase that needs them, not upfront.

• references/verification-protocol.md — the ladder in order, what to read off the project versus what to ask a human, what makes a step pass. Read at G3, before the UoW template is written: the decision it describes changes what goes into uow.md.
• references/config-schema.md — every key of the verify: block, and the credentials file
• references/login-recipes.md — the four auth presets, their selectors and their failure modes
• references/templates.md — 07-verification.md, the uow.md block, 08-evidence.md, the PR draft
• scripts/verify.py — the ladder, the run, the generated artifacts. Stdlib only.
• scripts/evidence_check.py — turns a ticked checkbox back into a checkable claim. Stdlib only.
• scripts/runner/run.py — the only file with a dependency; drives Playwright, writes run.json

Notes

• Responsive is a per-feature choice, not a global one. 07-verification.md declares viewports:. A desktop-only slice declares [desktop] and the mobile requirement stops applying to it — better than every feature carrying a mobile checkbox that gets ticked without meaning.
• Never hardcode tenant or account ids in step paths. The runner reads whatever ids the app resolves to after login off the landing URL and exposes them as {placeholders}.
• A cold start is not a flaky test. Each environment gets one warm-up request that no verdict is derived from, before the session is established — so a load balancer scaling from zero shows up as a warm-up duration in the evidence rather than as an intermittent step. The distinction is load-bearing: a retry budget around steps would hide a real regression by the same mechanism it hides a slow environment. warmup.attempts is capped at 5 to keep the two from merging.
• Credentials are read, never written or echoed. --doctor reports whether a value is present, never what it is, and no credential reaches run.json, the report, or the PR draft. The run plan goes to the runner over stdin, so it never lands on disk.
• Prod defaults to off. Turning it on is a deliberate edit, and writes: false is the posture that belongs there.