docs-live-test

name

docs-live-test

description

Verify a docs guide actually works by building a fixture workspace in the guide's "before" state, then having a fresh agent follow ONLY the page to reach the guide's success criterion. Reports page gaps/mismatches and patches the page. Use when asked to "live test this guide", "verify the page works", "test the migration doc", or before shipping any migration/setup guide. Proven on the ESLint flat-config guide (DOC-549): found 2 page gaps + 2 generator bugs.

Docs Live Test

Test a documentation guide the way a reader experiences it: no author knowledge, just the page.

Inputs

• Page under test (file path in the repo, e.g. astro-docs/src/content/docs/...)
• The "before" state the guide assumes (e.g. legacy configs, old versions)
• Success criterion (e.g. nx run-many -t lint exits 0, app builds, feature works)

Process

1. Setup agent (background, general-purpose)

Builds the fixture in the session scratchpad (NEVER inside the nx repo - gitignore kills watchers and pollutes the worktree):

• Create a real workspace (create-nx-workspace or nx init; check --help, never guess flags)
• Regress it to the guide's "before" state, planting one artifact per case the page addresses (e.g. a JSON config, a JS config the tool skips, an ignore file, each problem rule)
• Confirm the success criterion FAILS and capture the exact error
• git commit the baseline; write FIXTURE_STATE.md (versions, planted cases, failure output)
• Forbid the setup agent from reading the page under test (keeps the fixture unbiased)

2. Migrator agent (fresh, background)

• ONLY instruction source = the page under test (read once). Explicitly forbid reading other docs, package sources, or migration files
• Must reach the success criterion, then commit
• Log a FINDING for anything the page didn't cover: blocker (couldn't proceed from page alone), gap (needed knowledge the page omits), mismatch (page says X, reality is Y), nit
• Structured final report: result, steps mapped to page sections, findings, end state per planted case

3. Verify + patch (main loop)

• Independently rerun the success criterion in the fixture (nx reset first) - do not trust the agent's claim
• Check every planted case resolved the way the page prescribes (renames applied, not dropped; files deleted; etc.)
• Patch the page for each gap/mismatch (prettier + vale + commit per house docs flow)
• Product bugs the test exposes (generator/CLI defects) are NOT docs fixes: report them and offer to file Linear issues
• Report to Jack: pass/fail, gaps patched, bugs found, fixture path (note scratchpad is ephemeral; offer to move it if worth keeping)

Gotchas (learned on DOC-549)

• Nx-inferred targets can mask the expected failure mode (e.g. eslintrc lint targets inject ESLINT_USE_FLAT_CONFIG=false, so failure = missing rules, not "config not found"). Capture BOTH the wired and raw-tool failure in FIXTURE_STATE.md and check the page's claims against the wired one.
• Pin the tool major the guide targets (the fixture once needed ESLint 9, not the latest 10).
• The migrator inevitably knows things - the findings log is the firewall: anything it did that the page didn't say is a gap, even when it succeeded.