| name | quality-playbook |
| description | Run a complete quality engineering audit on any codebase. Derives behavioral requirements from the code, generates spec-traced functional tests, runs a three-pass code review with regression tests, executes a multi-model spec audit (Council of Three), and produces a consolidated bug report with TDD-verified patches. Finds the 35% of real defects that structural code review alone cannot catch. Works with any language. Trigger on 'quality playbook', 'spec audit', 'Council of Three', 'fitness-to-purpose', or 'coverage theater'. |
| license | Complete terms in LICENSE.txt |
| metadata | {"version":"1.5.3","author":"Andrew Stellman","github":"https://github.com/andrewstellman/quality-playbook"} |
Quality Playbook Generator
Plan Overview — read this first, then explain it to the user
Before reading any other section of this skill, understand the plan and its dependencies. Each phase produces artifacts that the next phase depends on. Skipping or rushing a phase means every downstream phase works from incomplete information.
Phase 0 (Prior Run Analysis): If previous quality runs exist, load their findings as seed data. This is automatic and only applies to re-runs.
Phase 1 (Explore): Run the v1.5.3 documentation intake first (python -m bin.reference_docs_ingest <target> to walk reference_docs/ — cite/ files produce quality/formal_docs_manifest.json records; top-level files are loaded as Tier 4 context via reference_docs_ingest.load_tier4_context(<target>)). Then explore the codebase in three stages: open exploration driven by domain knowledge, domain-knowledge risk analysis, and selected structured exploration patterns. Write all findings to quality/EXPLORATION.md. This file is the foundation — Phase 2 reads it as its primary input.
Phase 2 (Generate): Read EXPLORATION.md and produce the quality artifacts: requirements, constitution, functional tests, code review protocol, integration tests, spec audit protocol, TDD protocol, AGENTS.md.
Phase 3 (Code Review): Run the three-pass code review against HEAD. Write regression tests for every confirmed bug. Generate patches.
Phase 4 (Spec Audit): Three independent AI auditors review the code against requirements. Triage with verification probes. After triage, the same Council runs the v1.5.3 Layer-2 semantic citation check — one prompt per reviewer, structured per-REQ verdicts for every Tier 1/2 citation, output to quality/citation_semantic_check.json. Write regression tests for net-new findings.
Phase 5 (Reconciliation): Close the loop — every bug from code review and spec audit is tracked, regression-tested or explicitly exempted. Run TDD red-green cycle. Finalize the completeness report.
Phase 6 (Verify): Run self-check benchmarks against all generated artifacts. Check for internal consistency, version stamp correctness, and convergence.
Phase 7 (Present, Explore, Improve): Present results to the user with a scannable summary table, offer drill-down on any artifact, and provide a menu of improvement paths (iteration strategies, requirement refinement, integration test tuning). This is the interactive phase where the user takes ownership of the quality system.
Every bug found traces back to a requirement, and every requirement traces back to an exploration finding.
The critical dependency chain: Exploration findings → EXPLORATION.md → Requirements → Code review + Spec audit → Bug discovery. A shallow exploration produces abstract requirements. Abstract requirements miss bugs. The exploration phase is where bugs are won or lost.
MANDATORY FIRST ACTION: After reading and understanding the plan above, print the following message to the user, then explain the plan in your own words — what you'll do, what each phase produces, and why the exploration phase matters most. Emphasize that exploration starts with open-ended domain-driven investigation, followed by domain-knowledge risk analysis that reasons about what goes wrong in systems like this, then supplemented by selected structured patterns. Do not copy the plan verbatim; paraphrase it to demonstrate understanding.
Quality Playbook v1.5.3 — by Andrew Stellman
https://github.com/andrewstellman/quality-playbook
Generate a complete quality system tailored to a specific codebase. Unlike test stub generators that work mechanically from source code, this skill explores the project first — understanding its domain, architecture, specifications, and failure history — then produces a quality playbook grounded in what it finds.
Locating reference files
This skill references files in a references/ directory (e.g., references/iteration.md, references/review_protocols.md). The location depends on how the skill was installed. When a reference file is mentioned, resolve it by checking these paths in order and using the first one that exists:
references/ (relative to SKILL.md — works when running from the skill directory).claude/skills/quality-playbook/references/ (Claude Code installation).github/skills/references/ (GitHub Copilot flat installation).github/skills/quality-playbook/references/ (alternate Copilot installation)
All reference file mentions in this skill use the short form references/filename.md. If the relative path doesn't resolve, walk the fallback list above.
Why This Exists
Most software projects have tests, but few have a quality system. Tests check whether code works. A quality system answers harder questions: what does "working correctly" mean for this specific project? What are the ways it could fail that wouldn't be caught by tests? What should every developer (human or AI) know before touching this code?
Without a quality playbook, every new contributor (and every new AI session) starts from scratch — guessing at what matters, writing tests that look good but don't catch real bugs, and rediscovering failure modes that were already found and fixed months ago. A quality playbook makes the bar explicit, persistent, and inherited.
What This Skill Produces
Nine files that together form a repeatable quality system:
| File | Purpose | Why It Matters | Executes Code? |
|---|
quality/QUALITY.md | Quality constitution — coverage targets, fitness-to-purpose scenarios, theater prevention | Every AI session reads this first. It tells them what "good enough" means so they don't guess. | No |
quality/REQUIREMENTS.md | Testable requirements with project overview, use cases, and narrative — generated by a five-phase pipeline (contract extraction → derivation → verification → completeness → narrative) | The foundation for Passes 2 and 3 of the code review. Without requirements, review is limited to structural anomalies (~65% ceiling). With them, the review can catch intent violations — absence bugs, cross-file contradictions, and design gaps that are invisible to code reading alone. | No |
quality/test_functional.* | Automated functional tests derived from specifications | The safety net. Tests tied to what the spec says should happen, not just what the code does. Use the project's language: test_functional.py (Python), FunctionalSpec.scala (Scala), functional.test.ts (TypeScript), FunctionalTest.java (Java), etc. | Yes |
quality/RUN_CODE_REVIEW.md | Three-pass code review protocol: structural review, requirement verification, cross-requirement consistency | Structural review alone misses ~35% of real defects. The three-pass pipeline adds requirement verification and consistency checking — backed by experiment evidence showing it finds bugs invisible to all structural review conditions. | No |
quality/RUN_INTEGRATION_TESTS.md | Integration test protocol — end-to-end pipeline across all variants | Unit tests pass, but does the system actually work end-to-end with real external services? | Yes |
quality/BUGS.md | Consolidated bug report with patches | Every confirmed bug in one place with reproduction details, spec basis, severity, and patch references. The single source of truth for what's broken and how to verify it. | No |
quality/RUN_TDD_TESTS.md | TDD red-green verification protocol | Proves each bug is real (test fails on unpatched code) and each fix works (test passes after patch). Stronger evidence than a bug report alone — maintainers trust FAIL→PASS demonstrations. | Yes |
quality/RUN_SPEC_AUDIT.md | Council of Three multi-model spec audit protocol | No single AI model catches everything. Three independent models with different blind spots catch defects that any one alone would miss. | No |
AGENTS.md | Bootstrap context for any AI session working on this project | The "read this first" file. Without it, AI sessions waste their first hour figuring out what's going on. | No |
Plus output directories: quality/code_reviews/, quality/spec_audits/, quality/results/, quality/history/.
The pipeline also generates supporting artifacts: quality/PROGRESS.md (phase-by-phase checkpoint log with cumulative BUG tracker), quality/CONTRACTS.md (behavioral contracts), quality/COVERAGE_MATRIX.md (traceability), quality/COMPLETENESS_REPORT.md (final gate), and quality/VERSION_HISTORY.md (review log). Phase 7 can additionally generate quality/REVIEW_REQUIREMENTS.md (interactive review protocol) and quality/REFINE_REQUIREMENTS.md (refinement pass protocol) for iterative improvement.
The two critical deliverables are the requirements file and the functional test file. The requirements file (quality/REQUIREMENTS.md) feeds the code review protocol's verification and consistency passes — it's what makes the code review catch more than structural anomalies. The functional test file (named for the project's language and test framework conventions) is the automated safety net. The Markdown protocols are documentation for humans and AI agents.
Complete Artifact Contract
The quality gate (quality_gate.py) validates these artifacts. If the gate checks for it, this skill must instruct its creation. This is the canonical list — any artifact not listed here should not be gate-enforced, and any gate check should trace to an artifact listed here.
| Artifact | Location | Required? | Created In |
|---|
| Formal docs manifest (v1.5.3) | quality/formal_docs_manifest.json | Yes | Phase 1 (bin/reference_docs_ingest.py) |
| Requirements manifest (v1.5.3) | quality/requirements_manifest.json | Yes | Phase 2 |
| Use cases manifest (v1.5.3) | quality/use_cases_manifest.json | Yes | Phase 2 |
| Bugs manifest (v1.5.3) | quality/bugs_manifest.json | If bugs found | Phase 3/4/5 |
| Citation semantic check (v1.5.3) | quality/citation_semantic_check.json | Yes | Phase 4 (Layer 2 Council) |
| Exploration findings | quality/EXPLORATION.md | Yes | Phase 1 |
| Quality constitution | quality/QUALITY.md | Yes | Phase 2 |
| Requirements (UC identifiers) | quality/REQUIREMENTS.md | Yes | Phase 2 |
| Behavioral contracts | quality/CONTRACTS.md | Yes | Phase 2 |
| Functional tests | quality/test_functional.* | Yes | Phase 2 |
| Regression tests | quality/test_regression.* | If bugs found | Phase 3 |
| Code review protocol | quality/RUN_CODE_REVIEW.md | Yes | Phase 2 |
| Integration test protocol | quality/RUN_INTEGRATION_TESTS.md | Yes | Phase 2 |
| Spec audit protocol | quality/RUN_SPEC_AUDIT.md | Yes | Phase 2 |
| TDD verification protocol | quality/RUN_TDD_TESTS.md | Yes | Phase 2 |
| Bug tracker | quality/BUGS.md | Yes | Phase 3 |
| Coverage matrix | quality/COVERAGE_MATRIX.md | Yes | Phase 2 |
| Completeness report | quality/COMPLETENESS_REPORT.md | Yes | Phase 2 (baseline), Phase 5 (final verdict) |
| Progress tracker | quality/PROGRESS.md | Yes | Throughout |
| AI bootstrap | AGENTS.md | Yes | Phase 2 |
| Bug writeups | quality/writeups/BUG-NNN.md | If bugs found | Phase 5 |
| Regression patches | quality/patches/BUG-NNN-regression-test.patch | If bugs found | Phase 3 |
| Fix patches | quality/patches/BUG-NNN-fix.patch | Optional | Phase 3 |
| TDD traceability | quality/TDD_TRACEABILITY.md | If bugs have red-phase results | Phase 5 |
| TDD sidecar | quality/results/tdd-results.json | If bugs found | Phase 5 |
| TDD red-phase logs | quality/results/BUG-NNN.red.log | If bugs found | Phase 5 |
| TDD green-phase logs | quality/results/BUG-NNN.green.log | If fix patch exists | Phase 5 |
| Integration sidecar | quality/results/integration-results.json | When integration tests run | Phase 5 |
| Mechanical verify script | quality/mechanical/verify.sh | Yes (benchmark) | Phase 2 |
| Verify receipt | quality/results/mechanical-verify.log + .exit | Yes (benchmark) | Phase 5 |
| Triage probes | quality/spec_audits/triage_probes.sh | When triage runs | Phase 4 |
| Code review reports | quality/code_reviews/*.md | Yes | Phase 3 |
| Spec audit reports | quality/spec_audits/*auditor*.md + *triage* | Yes | Phase 4 |
| Recheck results (JSON) | quality/results/recheck-results.json | When recheck runs | Recheck |
| Recheck summary (MD) | quality/results/recheck-summary.md | When recheck runs | Recheck |
| Seed checks | quality/SEED_CHECKS.md | If Phase 0b ran | Phase 0b |
| Run metadata | quality/results/run-YYYY-MM-DDTHH-MM-SS.json | Yes | Phase 1 (created), Throughout (updated) |
Sidecar JSON lifecycle: Write all bug writeups before finalizing tdd-results.json — the sidecar's writeup_path field must point to an existing file, not a placeholder. Similarly, run integration tests and collect results before writing integration-results.json.
Sidecar JSON Canonical Examples
quality/results/tdd-results.json — the gate validates field names, not just presence:
{
"schema_version": "1.1",
"skill_version": "1.5.3",
"date": "2026-04-12",
"project": "repo-name",
"bugs": [
{
"id": "BUG-001",
"requirement": "REQ-003",
"red_phase": "fail",
"green_phase": "pass",
"verdict": "TDD verified",
"fix_patch_present": true,
"writeup_path": "quality/writeups/BUG-001.md"
}
],
"summary": {
"total": 3, "confirmed_open": 1, "red_failed": 0, "green_failed": 0, "verified": 2
}
}
verdict must be one of: "TDD verified", "red failed", "green failed", "confirmed open", "deferred". date must be ISO 8601 (YYYY-MM-DD), not a placeholder, not in the future.
quality/results/integration-results.json:
{
"schema_version": "1.1",
"skill_version": "1.5.3",
"date": "2026-04-12",
"project": "repo-name",
"recommendation": "SHIP",
"groups": [{ "group": 1, "name": "Group 1", "use_cases": ["UC-01"], "result": "pass", "tests_passed": 3, "tests_failed": 0, "notes": "" }],
"summary": { "total_groups": 12, "passed": 11, "failed": 1, "skipped": 0 },
"uc_coverage": { "UC-01": "covered_pass", "UC-02": "not_mapped" }
}
recommendation must be one of: "SHIP", "FIX BEFORE MERGE", "BLOCK". uc_coverage maps UC identifiers from REQUIREMENTS.md to coverage status.
Run Metadata
Every playbook run creates a timestamped metadata file at quality/results/run-YYYY-MM-DDTHH-MM-SS.json. This enables multi-model comparison and run history tracking.
Lifecycle: Create this file at the start of Phase 1. Update phases_completed, bug_count, and end_time as each phase finishes. The final update happens after the terminal gate.
{
"schema_version": "1.0",
"skill_version": "1.5.3",
"project": "repo-name",
"model": "claude-sonnet-4-6",
"model_provider": "anthropic",
"runner": "claude-code",
"start_time": "2026-04-16T10:30:00Z",
"end_time": "2026-04-16T11:45:00Z",
"duration_minutes": 75,
"phases_completed": ["Phase 0b", "Phase 1", "Phase 2", "Phase 3", "Phase 4", "Phase 5"],
"iterations_completed": ["gap", "unfiltered", "parity", "adversarial"],
"bug_count": 12,
"bug_severity": { "HIGH": 2, "MEDIUM": 5, "LOW": 5 },
"gate_result": "PASS",
"gate_fail_count": 0,
"gate_warn_count": 2,
"notes": ""
}
Required fields: schema_version, skill_version, project, model, start_time. All other fields are populated as the run progresses. model should be the exact model string (e.g., "claude-sonnet-4-6", "gpt-4.1", "claude-opus-4-6"). runner identifies the tool used to execute the playbook (e.g., "claude-code", "copilot-cli", "cursor", "cowork"). duration_minutes is computed from end_time - start_time. If the model or runner cannot be determined, use "unknown".
How to Use
The playbook is designed to run one phase at a time. Each phase runs in its own session with a clean context window, producing files on disk that the next phase reads. This gives much better results than running all phases at once — each phase gets the full context window for deep analysis instead of competing for space with other phases.
Default behavior: run Phase 1 only. When someone says "run the quality playbook" or "execute the quality playbook," run Phase 1 (Explore) and stop. After Phase 1 completes, tell the user what happened and what to say next. The user drives each phase forward explicitly.
Interactive protocol — how to guide the user
After every phase and every iteration, STOP and print guidance. Use a # header so it's prominent in the chat. The guidance must include: what just happened (one line), what the key outputs are, and the exact prompt to continue. See the end-of-phase messages defined after each phase section below.
If the user says "keep going", "continue", "next phase", "next", or anything similar, run the next phase in sequence. If all phases are complete, suggest the first iteration strategy (gap). If an iteration just finished, suggest the next strategy in the recommended cycle.
If the user says "run all phases", "run everything", or "run the full pipeline", run all phases sequentially in a single session. This uses more context but some users prefer it.
If the user asks "help", "how does this work", "what is this", or any similar phrasing, respond with this explanation (adapt the wording naturally, don't copy verbatim):
The Quality Playbook finds bugs that structural code review alone can't catch — the 35% of real defects that require understanding what the code is supposed to do. It works phase by phase:
- Phase 1 (Explore): Understand the codebase — architecture, risks, failure modes, specifications
- Phase 2 (Generate): Produce quality artifacts — requirements, tests, review protocols
- Phase 3 (Code Review): Three-pass review with regression tests for every confirmed bug
- Phase 4 (Spec Audit): Three independent AI auditors check the code against requirements
- Phase 5 (Reconciliation): Close the loop — TDD red-green verification for every bug
- Phase 6 (Verify): Self-check benchmarks validate all generated artifacts
After the numbered phases complete, you can run iteration strategies (gap, unfiltered, parity, adversarial) to find additional bugs — iterations typically add 40-60% more confirmed bugs on top of the baseline.
The playbook works best when you provide documentation alongside the code — specs, API docs, design documents, community documentation. It also gets significantly better results when you run each phase separately rather than all at once.
To get started, say: "Run the quality playbook on this project."
If the user asks "what happened", "what's going on", "where are we", or "what should I do next", read quality/PROGRESS.md and give them a concise status update: which phases are complete, how many bugs found so far, and what the next step is.
Documentation warning
At the start of Phase 1, before exploring any code, check for documentation. Look for directories named docs/, docs_gathered/, doc/, documentation/, or any gathered documentation files. Also check if the user mentioned documentation in their prompt.
If no documentation is found, print this warning immediately (before proceeding):
Important: No project documentation found. The quality playbook works without documentation, but it finds significantly more bugs — and higher-confidence bugs — when you provide specs, API docs, design documents, or community documentation. In controlled experiments, documentation-enriched runs found different and better bugs than code-only baselines.
If you have documentation available, you can add it to a docs_gathered/ directory and re-run Phase 1. Otherwise, I'll proceed with code-only analysis.
Then proceed with Phase 1 — don't block on this, just make sure the user sees the warning.
Running a specific phase
The user can request any individual phase:
Run quality playbook phase 1.
Run quality playbook phase 3 — code review.
Run phase 5 reconciliation.
When running a specific phase, check that its prerequisites exist (e.g., Phase 3 requires Phase 2 artifacts). If prerequisites are missing, tell the user which phases need to run first.
Iteration mode — improve on a previous run
Use this when a previous playbook run exists and you want to find additional bugs. Iteration mode replaces Phase 1's from-scratch exploration with a targeted exploration using one of five strategies, then merges findings with the previous run and re-runs Phases 2–6 against the combined results.
When to use iteration mode: After a complete playbook run, when you believe the codebase has more bugs than the first run found. This is especially effective for large codebases where a single run can only cover 3–5 subsystems, and for library/framework codebases where different exploration paths find different bug classes.
Read references/iteration.md for detailed strategy instructions. That file contains the full operational detail for each strategy, shared rules, merge steps, and the completion gate. The summary below describes when to use each strategy.
TDD applies to iteration runs. Every newly confirmed bug in an iteration run must go through the full TDD red-green cycle and produce quality/results/BUG-NNN.red.log (and .green.log if a fix patch exists). The quality gate enforces this — missing logs cause FAIL. See references/iteration.md shared rule 5 and the TDD Log Closure Gate in Phase 5.
Iteration strategies. The user selects a strategy by naming it in the prompt. If no strategy is named, default to gap.
Run the next iteration of the quality playbook. # default: gap strategy
Run the next iteration of the quality playbook using the gap strategy.
Run the next iteration using the unfiltered strategy.
Run the next iteration using the parity strategy.
Run an iteration using the adversarial strategy.
Recommended cycle: gap → unfiltered → parity → adversarial. Each strategy finds different bug classes:
gap (default) — Scan previous coverage, explore uncovered subsystems and thin sections. Best when the first run was structurally sound but only covered a subset of the codebase.unfiltered — Pure domain-driven exploration with no structural constraints. No pattern templates, no applicability matrices, no section format requirements. Recovers bugs that structured exploration suppresses.parity — Systematically enumerate parallel implementations of the same contract (transport variants, fallback chains, setup-vs-reset paths) and diff them for inconsistencies. Finds bugs that only emerge from cross-path comparison.adversarial — Re-investigate dismissed/demoted triage findings and challenge thin SATISFIED verdicts. Recovers Type II errors from conservative triage.all — Runner-level convenience: executes gap → unfiltered → parity → adversarial in sequence, each as a separate agent session. Stops early if a strategy finds zero new bugs.
Phase-by-phase execution
Each phase produces files on disk that the next phase reads. This is how context transfers between phases — through files, not through conversation history. The key handoff files are:
quality/EXPLORATION.md — Phase 1 writes this, Phase 2 reads it. Contains everything Phase 2 needs to generate artifacts without re-exploring the codebase.quality/PROGRESS.md — Updated after every phase. Cumulative BUG tracker ensures no finding is lost.- Generated artifacts (REQUIREMENTS.md, CONTRACTS.md, etc.) — Phase 2 writes these, Phases 3–5 read them to run reviews, audits, and reconciliation.
The pattern for each phase boundary: finish the current phase, write everything to disk, then print the end-of-phase message and stop. When the user starts the next phase, read back the files you need before proceeding. This "write then read" cycle is the phase boundary — it lets you drop exploration context from working memory before loading review context, for example.
Write your Phase 1 exploration findings to quality/EXPLORATION.md before proceeding. This file is mandatory in all modes. Make it thorough: domain identification, architecture map, existing tests, specification summary, quality risks, skeleton/dispatch analysis, derived requirements (REQ-NNN), and derived use cases (UC-NN). Everything Phase 2 needs to generate artifacts must be in this file.
The discipline of writing exploration findings to disk is what forces thorough analysis. Without it, the model keeps vague impressions in working memory and produces broad, abstract requirements that miss function-level defects. Writing forces specificity: file paths, line numbers, exact function names, concrete behavioral rules. That specificity is what makes requirements precise enough to catch bugs during code review.
Phase 0: Prior Run Analysis (Automatic)
This phase runs only if quality/runs/ exists and contains prior quality artifacts. If there are no prior runs, skip to Phase 1. If quality/runs/ exists but is empty or contains no conformant quality artifacts (no subdirectories with quality/BUGS.md under them), skip Phase 0a and fall through to Phase 0b.
When prior runs exist, the playbook enters continuation mode. This enables iterative bug discovery: each run inherits confirmed findings from prior runs, verifies them mechanically, and explores for additional bugs. The iteration converges when a run finds zero net-new bugs.
Step 0a: Build the seed list. Read quality/runs/*/quality/BUGS.md from all prior runs. For each confirmed bug, extract: bug ID, file:line, summary, and the regression test assertion. Deduplicate by file:line (the same bug found in multiple runs counts once). Write the merged seed list to quality/SEED_CHECKS.md with this format:
## Seed Checks (from N prior runs)
| Seed | Origin Run | File:Line | Summary | Assertion |
|------|-----------|-----------|---------|-----------|
| SEED-001 | run-1 | virtio_ring.c:3509-3529 | RING_RESET dropped | `"case VIRTIO_F_RING_RESET:" in func` |
Step 0b: Execute seed checks mechanically. For each seed, run the assertion against the current source tree. Record PASS (bug was fixed since last run) or FAIL (bug still present). A failing seed is a confirmed carry-forward bug — it must appear in this run's BUGS.md regardless of whether any auditor independently finds it. A passing seed means the bug was fixed — note it in PROGRESS.md as "SEED-NNN: resolved since prior run."
Step 0c: Identify prior-run scope. Read quality/runs/*/quality/PROGRESS.md for scope declarations. Note which subsystems were covered in prior runs. During Phase 1 exploration, prioritize areas NOT covered by prior runs to maximize the chance of finding new bugs. If all subsystems were covered in prior runs, explore the same scope but with different emphasis (e.g., different scrutiny areas, different entry points).
Step 0d: Inject seeds into downstream phases. The seed list becomes input to:
- Phase 3 (code review): Add to the code review prompt: "Prior runs confirmed these bugs — verify they are still present and look for additional findings in the same subsystems."
- Phase 4 (spec audit): Add to
RUN_SPEC_AUDIT.md: "Known open issues from prior runs: [seed list]. Expect auditors to find these. If an auditor does NOT flag a known seed bug, that is a coverage gap in their review, not evidence the bug was fixed."
Why this exists: Non-deterministic scope exploration means different runs notice different bugs. In cross-version testing, 4/8 repos had bugs found in some versions but not others — not because the bugs were fixed, but because the model explored different parts of the codebase. Iterating with seed injection solves this: confirmed bugs carry forward mechanically (no re-discovery needed), and each new run can focus exploration on uncovered territory.
Phase 0b: Sibling-Run Seed Discovery (Automatic)
This step runs only if quality/runs/ does not exist OR quality/runs/ exists but contains no conformant quality artifacts (i.e., Phase 0a has nothing to work with) and the project directory is versioned (e.g., httpx-1.3.23/ sits alongside httpx-1.3.21/). If quality/runs/ exists with conformant artifacts, Phase 0a already handles seed injection — skip this step.
If quality/runs/ exists but is empty or contains only non-conformant subdirectories, emit a warning: "Phase 0b: quality/runs/ exists but contains no conformant artifacts — consulting sibling versioned directories for seeds." Then proceed with the sibling discovery below.
When no quality/runs/ directory exists but sibling versioned directories do, look for prior quality artifacts in those siblings:
- Discover siblings. List directories matching the pattern
<project-name>-<version>/quality/BUGS.md relative to the parent directory. Exclude the current directory. Sort by version descending (most recent first).
- Import confirmed bugs as seeds. For each sibling with a
quality/BUGS.md, extract confirmed bugs using the same format as Step 0a. Write them to quality/SEED_CHECKS.md with origin noted as the sibling directory name.
- Execute seed checks mechanically (same as Step 0b in Phase 0a). For each imported seed, run the assertion against the current source tree and record PASS/FAIL.
- Inject into downstream phases (same as Step 0d in Phase 0a).
Why this exists: In v1.3.23 benchmarking, httpx produced a zero-bug result despite httpx-1.3.21 having found the Headers.__setitem__ non-ASCII encoding bug. The model simply explored different code paths and never examined the Headers area. Sibling-run seeding ensures that bugs confirmed in prior versioned runs carry forward even without an explicit quality/runs/ archive. This is a different failure class than mechanical tampering — it addresses exploration non-determinism, not evidence corruption.
Phase 1: Explore the Codebase (Write As You Go)
Required references for this phase — read these before proceeding:
references/exploration_patterns.md — six bug-finding patterns to apply after open exploration
First action: create run metadata. Before any exploration, create the run metadata file:
mkdir -p quality/results
cat > "quality/results/run-$(date -u +%Y-%m-%dT%H-%M-%S).json" <<'METADATA'
{
"schema_version": "1.0",
"skill_version": "1.5.3",
"project": "<repo-name>",
"model": "<model-string>",
"model_provider": "<provider>",
"runner": "<tool>",
"start_time": "<ISO-8601-UTC>",
"end_time": null,
"duration_minutes": null,
"phases_completed": [],
"iterations_completed": [],
"bug_count": 0,
"bug_severity": { "HIGH": 0, "MEDIUM": 0, "LOW": 0 },
"gate_result": null,
"gate_fail_count": null,
"gate_warn_count": null,
"notes": ""
}
METADATA
Fill in project, model (exact model string, e.g., "claude-sonnet-4-6"), model_provider (e.g., "anthropic", "openai", "cursor"), runner (e.g., "claude-code", "copilot-cli", "cursor"), and start_time (UTC ISO 8601). Update this file at the end of each phase — append the completed phase to phases_completed and update bug_count/bug_severity as bugs are confirmed. The final update after the terminal gate fills in end_time, duration_minutes, and gate_result.
Second action: run v1.5.3 document ingest (before exploring any code). A single stdlib-only module in bin/ produces the authoritative documentation record that Phase 1 requirement derivation depends on:
python -m bin.reference_docs_ingest <target> — walks reference_docs/ in the target repo once. Files under reference_docs/cite/ are hashed and written to quality/formal_docs_manifest.json per schemas.md §4 and the §1.6 manifest wrapper. Files at the top level of reference_docs/ are not written to the manifest but are available as Tier 4 context via bin.reference_docs_ingest.load_tier4_context(<target>), which returns a sorted list of (path, text) tuples. If the ingest command fails (unsupported extension, non-UTF-8 bytes), stop the run and surface the stderr output to the user verbatim — ingest errors are actionable and must be fixed before exploration continues.
No sidecar needed. Folder placement is the flag: top-level reference_docs/<name>.<ext> files are Tier 4 context; files under reference_docs/cite/<name>.<ext> are citable sources. Tier 1 is the default for cite/ contents; a file may override to Tier 2 with an optional in-file marker on the first non-blank line: <!-- qpb-tier: 2 --> (Markdown) or # qpb-tier: 2 (plaintext). README.md under either folder is skipped.
When reference_docs/ is missing or empty, Phase 1 MUST print this actionable message and proceed:
Phase 1 found no documentation in reference_docs/. The playbook will proceed
using only Tier 3 evidence (the source tree itself). For better results, drop
plaintext documentation into:
reference_docs/ ← AI chats, design notes, retrospectives (Tier 4 context)
reference_docs/cite/ ← project specs, RFCs, API contracts (citable, byte-verified)
See README.md "Step 1: Provide documentation" for details.
Plaintext only — conversion happens outside the playbook. Reference docs are .txt or .md only (schemas.md §2). PDFs, DOCX, HTML, etc. are rejected with an actionable conversion hint (pdftotext, pandoc -t plain, lynx -dump). Do NOT attempt to parse binary or formatted documents inside the skill — run the conversion outside and commit the plaintext.
Spend the first phase understanding the project. The quality playbook must be grounded in this specific codebase — not generic advice.
Why explore first? The most common failure in AI-generated quality playbooks is producing generic content — coverage targets that could apply to any project, scenarios that describe theoretical failures, tests that exercise language builtins instead of project code. Exploration prevents this by forcing every output to reference something real: a specific function, a specific schema, a specific defensive code pattern. If you can't point to where something lives in the code, you're guessing — and guesses produce quality playbooks nobody trusts.
Scaling for large codebases: For projects with more than ~50 source files, don't try to read everything. Focus exploration on the 3–5 core modules (the ones that handle the primary data flow, the most complex logic, and the most failure-prone operations). Read representative tests from each subsystem rather than every test file. The goal is depth on what matters, not breadth across everything.
Depth over breadth (critical). A narrow scope with function-level detail finds more bugs than a broad scope with subsystem-level summaries. For each core module you explore, identify the specific functions that implement critical behavior and document them by name, file path, and line number. Requirements derived from "the reset subsystem should handle errors" will not catch bugs. Requirements derived from "vm_reset() at virtio_mmio.c:256 must poll the status register after writing zero" will. The difference between a useful exploration and a useless one is specificity — file paths, function names, line numbers, exact behavioral rules.
Three-stage exploration: open first, then domain risks, then selected patterns. Exploration has three stages, and the order matters:
-
Open exploration (domain-driven). Before applying any structured pattern, explore the codebase the way an experienced developer would: read the code, understand the architecture, identify risks based on your domain knowledge of what goes wrong in systems like this one. Ask yourself: "What would an expert in [this domain] check first?" For an HTTP library, that means redirect handling, header encoding, connection lifecycle. For a CLI framework, that means flag parsing, help generation, completion/validation consistency. For a serialization library, that means type coverage, round-trip fidelity, edge-case handling. Write concrete findings with file paths and line numbers. This stage must produce at least 8 concrete bug hypotheses or suspicious findings — not architectural observations, but specific "this code at file:line might be wrong because [reason]" findings. At least 4 must reference different modules or subsystems.
-
Domain-knowledge risk analysis. After open exploration, step back from the code and reason about what you know from training about systems like this one. This is the primary bug-hunting pass for library and framework codebases. Complete the Step 6 questions below using two sources — the code you just explored AND your domain knowledge of similar systems. Generate at least 5 ranked failure scenarios, each naming a specific function, file, and line, and explaining why a domain-specific edge case produces wrong behavior. You don't need to have observed these failures — you know from training that they happen to systems of this type. Write the results to the ## Quality Risks section of EXPLORATION.md before proceeding to patterns.
What this stage must NOT produce: A section that lists defensive patterns the code already has (things the code does RIGHT) is not a risk analysis. A section that lists risky modules without specific failure scenarios is not a risk analysis. A section that concludes "this is a mature, well-tested library so basic bugs are unlikely" is actively harmful — mature libraries have the most subtle bugs, precisely because the obvious ones were found years ago. The test: could a code reviewer read each scenario and immediately know what to check? If not, the scenario is too abstract.
-
Pattern-driven exploration (selected, not exhaustive). After open exploration and domain-risk analysis are written to disk, evaluate all six analysis patterns from exploration_patterns.md using a pattern applicability matrix. For each pattern, assess whether it applies to this codebase and what it would target. Then select 3 to 4 patterns for deep-dive treatment — the highest-yield patterns for this specific codebase. The remaining patterns get a brief "not applicable" or "deferred" note with codebase-specific rationale. Do not produce deep sections for all six patterns — depth on 3–4 beats shallow coverage of 6. Select 4 when a fourth pattern has clear applicability and would cover code areas not reached by the other three; default to 3 when in doubt.
For each selected pattern deep dive, use the output format from the reference file and trace code paths across 2+ functions. The deep dives should pressure-test, refine, or extend the findings from the open exploration and risk analysis — not repeat them.
The Phase 1 completion gate checks for all three stages. The open exploration section, the quality risks section, the pattern applicability matrix, and the pattern deep-dive sections must all be present.
Write incrementally — do not hold findings in memory. This is the single most important execution rule in Phase 1. After you explore each subsystem or apply each pattern, immediately append your findings to quality/EXPLORATION.md on disk before moving to the next subsystem or pattern. Do not try to hold findings in working memory across multiple subsystems. The write-as-you-go discipline serves two purposes:
-
Depth recovery. If you explore the PCI interrupt routing subsystem and find suspicious code at vp_find_vqs_intx(), write that finding to EXPLORATION.md immediately. Then when you move to the admin queue subsystem, your working memory is free to go deep there. Without incremental writes, findings from the first subsystem compete with findings from the second, and both end up shallow.
-
Nothing gets lost. In v1.3.41 benchmarking, the model explored 8 pattern sections but wrote only 5–7 lines per section — perfectly uniform, perfectly shallow. Every section passed the gate but none went deep enough to find bugs that require tracing code paths across multiple functions. The model was trying to compose the entire EXPLORATION.md at the end, after reading everything, and could only recall the surface-level findings. Incremental writes prevent this.
The rhythm is: read a subsystem → write findings to disk → read the next subsystem → append findings → repeat. Each append should include specific function names, file paths, line numbers, and concrete bug hypotheses. A 5-line section that says "checked cross-implementation consistency, found one gap" is a gate-passing placeholder, not an exploration finding. A useful section traces a code path: "function A at file:line calls function B at file:line, which does X but not Y; compare with function C at file:line which does both X and Y."
Mandatory consolidation step. After all three stages (open exploration, quality risks, and selected pattern deep dives) are explored and written to EXPLORATION.md, add a final section: ## Candidate Bugs for Phase 2. This section consolidates the strongest bug hypotheses from all earlier sections into a prioritized handoff list. For each candidate, include: the hypothesis, the specific file:line references, which stage surfaced it (open exploration, quality risks, or pattern), and what the code review should look for. This section is the bridge between exploration and artifact generation — it tells Phase 3 exactly where to focus. Minimum: 4 candidate bugs with file:line references — at least 2 from open exploration or quality risks, and at least 1 from a pattern deep dive. There is no maximum.
Pre-flight: Scope declaration for large repositories
Before exploring any source code, estimate scale: approximate source-file count (excluding tests, docs, and generated files), major subsystem count, and documentation volume. Note the count in PROGRESS.md.
- Fewer than 200 source files: Proceed with full exploration. The depth-vs-breadth guidance above still applies.
- 200–500 source files: Declare your intended scope before exploring. Write a
## Scope declaration section to PROGRESS.md naming the 3–5 subsystems you will cover, the expected file count for each, and which subsystems you are deferring with rationale. Then proceed with exploration of the declared scope only.
- More than 500 source files: Stop and write a mandatory scope declaration to PROGRESS.md before reading any source files. The scope declaration must include: (a) the subsystems covered in this run, (b) the subsystems explicitly deferred, (c) the exclusion rationale for each deferred subsystem, and (d) recommended subsystem scope for follow-on runs. Do not begin exploration until this is written. A scope declaration that covers "everything" is not valid for repositories above this threshold.
Resuming a previous session: If PROGRESS.md already exists and shows phases marked complete, read it first. Do not redo phases already marked complete — resume from the first phase marked incomplete. If a scope declaration is already written, honor it exactly. If the previous session's scope declaration deferred subsystems, do not expand scope to cover them unless this run is explicitly a follow-on for the deferred areas.
Specification-primary repositories: Some repositories ship a specification, configuration, or protocol document as their primary product, with executable code as supporting infrastructure. Examples: a skill definition with benchmark tooling, a schema registry with validation scripts, a pipeline config with orchestration helpers. When the primary product is a specification rather than executable code, derive requirements from the specification's internal consistency, completeness, and correctness — not just from the executable code paths. The specification is the thing users depend on; the tooling is secondary. If you find yourself writing 80%+ of requirements about helper scripts and <20% about the primary specification, you have the focus inverted.
Step 0: Ask About Development History
Before exploring code, ask the user one question:
"Do you have exported AI chat history from developing this project — Claude exports, Gemini takeouts, ChatGPT exports, Claude Code transcripts, or similar? If so, point me to the folder. The design discussions, incident reports, and quality decisions in those chats will make the generated quality playbook significantly better."
If the user provides a chat history folder:
- Scan for an index file first. Look for files named
INDEX*, CONTEXT.md, README.md, or similar navigation aids. If one exists, read it — it will tell you what's there and how to find things.
- Search for quality-relevant conversations. Look for messages mentioning: quality, testing, coverage, bugs, failures, incidents, crashes, validation, retry, recovery, spec, fitness, audit, review. Also search for the project name.
- Extract design decisions and incident history. The most valuable content is: (a) incident reports — what went wrong, how many records affected, how it was detected, (b) design discussions — why a particular approach was chosen, what alternatives were rejected, (c) quality framework discussions — coverage targets, testing philosophy, model review experiences, (d) cross-model feedback — where different AI models disagreed about the code.
- Don't try to read everything. Chat histories can be enormous. Use the index to find the most relevant conversations, then search within those for quality-related content. 10 minutes of targeted searching beats 2 hours of exhaustive reading.
This context is gold. A chat history where the developer discussed "why we chose this concurrency model" or "the time we lost 1,693 records in production" transforms generic scenarios into authoritative ones.
If the user doesn't have chat history, proceed normally — the skill works without it, just with less context.
Autonomous fallback: When running in benchmark mode, via bin/run_playbook.py (benchmark runner, not shipped with the skill), or without user interaction (e.g., --single-pass), skip Step 0's question and proceed directly to Step 1. If chat history folders are visible in the project tree (e.g., AI Chat History/, .chat_exports/), scan them without asking. If no chat history is found, proceed — do not block waiting for a response that won't come.
Step 1: Identify Domain, Stack, and Specifications
Read the README, existing documentation, and build config (pyproject.toml / package.json / Cargo.toml). Answer:
- What does this project do? (One sentence.)
- What language and key dependencies?
- What external systems does it talk to?
- What is the primary output?
Find the specifications. Specs are the source of truth for functional tests. Search in order: AGENTS.md/CLAUDE.md in root, specs/, docs/, spec/, design/, architecture/, adr/, then .md files in root. Record the paths.
If no formal spec documents exist, the skill still works — but you need to assemble requirements from other sources. In order of preference:
- Ask the user — they often know the requirements even if they're not written down.
- README and inline documentation — many projects embed requirements in their README, API docs, or code comments.
- Existing test suite — tests are implicit specifications. If a test asserts
process(x) == y, that's a requirement.
- Type signatures and validation rules — schemas, type annotations, and validators define what the system accepts and rejects.
- Infer from code behavior — as a last resort, read the code and infer what it's supposed to do. Mark these as inferred requirements in QUALITY.md and flag them for user confirmation.
When working from non-formal requirements, label each scenario and test with a requirement tag that includes a confidence tier and source:
[Req: formal — README §3] — written by humans in a spec document. Authoritative.[Req: user-confirmed — "must handle empty input"] — stated by the user but not in a formal doc. Treat as authoritative.[Req: inferred — from validate_input() behavior] — deduced from code. Flag for user review.
Use this exact tag format in QUALITY.md scenarios, functional test documentation, and spec audit findings. It makes clear which requirements are authoritative and which need validation.
Step 1b: Evaluate Documentation Depth
If docs_gathered/ exists, read every file in it before deciding which subsystems to focus on. For each document, classify its depth:
- Deep — contains internal contracts, safety invariants, concurrency models, defensive patterns, error handling details, or line-number-level source references. Suitable for deriving requirements.
- Moderate — covers architecture and API surface with some implementation detail. Useful for orientation but insufficient alone for requirement derivation.
- Shallow — API catalog, feature overview, or marketing-level summary. Lists what exists but not how it works, how it fails, or what contracts it enforces. Not sufficient for scoping decisions.
The scoping rule: Do not narrow the audit scope to only the subsystems that have deep documentation. If the most complex or most failure-prone module has only shallow documentation, that is a documentation gap to flag in PROGRESS.md, not a reason to skip the module. The highest-risk code with the thinnest documentation is where bugs hide — auditing only well-documented areas produces a safe-looking report that misses real defects.
When documentation is shallow for a high-risk area:
- Note the gap explicitly in PROGRESS.md under a
## Documentation depth assessment section.
- Derive requirements from source code directly (doc comments, safety annotations, defensive patterns, existing tests) and tag them as
[Req: inferred — from source].
- Flag the area for deeper documentation gathering in the completeness report.
Record the depth classification for each docs_gathered/ file in PROGRESS.md so reviewers can assess whether the documentation influenced the scope appropriately.
Coverage commitment table: After classifying all docs_gathered/ documents, produce this table in PROGRESS.md under the ## Documentation depth assessment section:
| Document | Depth | Subsystem | Requirements commitment | If excluded: justification |
|---|
For every deep document, map it to the subsystem it covers, then either commit to deriving requirements from it ("will cover in Phase 2") or provide a specific justification that names the tradeoff. A sentence like "out of scope for this run" is not sufficient — the justification must say why, e.g., "interpreter JIT is excluded because this run focuses on the parser/compiler/GC pipeline; separate run recommended."
Gate: A high-risk subsystem documented deeply in docs_gathered/ must not silently disappear from the requirements set. If a deep document has a "will cover" commitment but produces zero requirements by the end of Step 7, the requirements pipeline is incomplete — go back and derive requirements for the gap before proceeding to Phase 2 artifact generation.
Step 2: Map the Architecture
List source directories and their purposes. Read the main entry point, trace execution flow. Identify:
- The 3–5 major subsystems
- The data flow (Input → Processing → Output)
- The most complex module
- The most fragile module
Step 3: Read Existing Tests
Read the existing test files — all of them for small/medium projects, or a representative sample from each subsystem for large ones. Identify: test count, coverage patterns, gaps, and any coverage theater (tests that look good but don't catch real bugs).
Critical: Record the import pattern. How do existing tests import project modules? Every language has its own conventions (Python sys.path manipulation, Java/Scala package imports, TypeScript relative paths or aliases, Go package/module paths, Rust use crate:: or use myproject::). You must use the exact same pattern in your functional tests — getting this wrong means every test fails with import/resolution errors. See references/functional_tests.md § "Import Pattern" for the full six-language matrix.
Identify integration test runners. Look for scripts or test files that exercise the system end-to-end against real external services (APIs, databases, etc.). Note their patterns — you'll need them for RUN_INTEGRATION_TESTS.md.
Step 4: Read the Specifications
Walk each spec document section by section. For every section, ask: "What testable requirement does this state?" Record spec requirements without corresponding tests — these are the gaps the functional tests must close.
If using inferred requirements (from tests, types, or code behavior), tag each with its confidence tier using the [Req: tier — source] format defined in Step 1. Inferred requirements feed into QUALITY.md scenarios and should be flagged for user review in Phase 7.
Step 4b: Read Function Signatures and Real Data
Before writing any test, you must know exactly how each function is called. For every module you identified in Step 2:
- Read the actual function signatures — parameter names, types, defaults. Don't guess from usage context — read the function definition and any documentation (Python docstrings, Java/Scala Javadoc/ScalaDoc, TypeScript type annotations, Go godoc comments, Rust doc comments and type signatures).
- Read real data files — If the project has items files, fixture files, config files, or sample data (in
pipelines/, fixtures/, test_data/, examples/), read them. Your test fixtures must match the real data shape exactly.
- Read existing test fixtures — How do existing tests create test data? Copy their patterns. If they build config dicts with specific keys, use those exact keys.
- Check library versions — Check the project's dependency manifest (
requirements.txt, build.sbt, package.json, pom.xml/build.gradle, go.mod, Cargo.toml) to see what's actually available. Don't write tests that depend on library features that aren't installed. If a dependency might be missing, use the test framework's skip mechanism — see references/functional_tests.md § "Library version awareness" for framework-specific examples.
Record a function call map: for each function you plan to test, write down its name, module, parameters, and what it returns. This map prevents the most common test failure: calling functions with wrong arguments.
Step 5: Find the Skeletons
This is the most important step. Search for defensive code patterns — each one is evidence of a past failure or known risk.
Why this matters: Developers don't write try/except blocks, null checks, or retry logic for fun. Every piece of defensive code exists because someone got burned. A try/except around a JSON parse means malformed JSON happened in production. A null check on a field means that field was missing when it shouldn't have been. These patterns are the codebase whispering its history of failures. Each one becomes a fitness-to-purpose scenario and a boundary test.
Read references/defensive_patterns.md for the systematic search approach, grep patterns, and how to convert findings into fitness-to-purpose scenarios and boundary tests.
Minimum bar: at least 2–3 defensive patterns per core source file. If you find fewer, you're skimming — read function bodies, not just signatures.
Step 5a: Trace State Machines
If the project has any kind of state management — status fields, lifecycle phases, workflow stages, mode flags — trace the state machine completely. This catches a category of bugs that defensive pattern analysis alone misses: states that exist but aren't handled.
How to find state machines: Search for status/state fields in models, enums, or constants (e.g., status, state, phase, mode). Search for guards that check status before allowing actions (e.g., if status == "running", match self.state). Search for state transitions (assignments to status fields).
For each state machine you find:
- Enumerate all possible states. Read the enum, the constants, or grep for every value the field is assigned. List them all.
- For each consumer of state (UI handlers, API endpoints, control flow guards), check: does it handle every possible state? A
switch/match without a meaningful default, or an if/elif chain that doesn't cover all states, is a gap.
- For each state transition, check: can you reach every state? Are there states you can enter but never leave? Are there states that block operations that should be available?
- Record gaps as findings. A status guard that allows action X for "running" but not for "stuck" is a real bug if the user needs to perform action X on stuck processes. A process that enters a terminal state but never triggers cleanup is a real bug.
Why this matters: State machine gaps produce bugs that are invisible during normal operation but surface under stress or edge conditions — exactly when you need the system to work. A batch processor that can't be killed when it's in "stuck" status, or a watcher that never self-terminates after all work completes, or a UI that refuses to resume a "pending" run, are all symptoms of incomplete state handling. These bugs don't show up in defensive pattern analysis because the code isn't defending against them — it's simply not handling them at all.
Step 5b: Map Schema Types
If the project has a validation layer (Pydantic models in Python, JSON Schema, TypeScript interfaces/Zod schemas, Java Bean Validation annotations, Scala case class codecs), read the schema definitions now. For every field you found a defensive pattern for, record what the schema accepts vs. rejects.
Read references/schema_mapping.md for the mapping format and why this matters for writing valid boundary tests.
Step 6: Domain-Knowledge Risk Analysis (Code + Domain Knowledge)
This is the primary bug-hunting pass for library and framework codebases. Complete it before selecting any structured patterns. Write the results to the ## Quality Risks section of EXPLORATION.md immediately — do not hold them in memory.
Every project has a different failure profile. This step uses two sources — not just code exploration, but your training knowledge of what goes wrong in similar systems.
From code exploration, ask:
- What does "silently wrong" look like for this project?
- What external dependencies can change without warning?
- What looks simple but is actually complex?
- Where do cross-cutting concerns hide?
From domain knowledge, ask:
- "What goes wrong in systems like this?" — If it's an HTTP router, think about header parsing edge cases (quality values, token lists, case sensitivity), middleware ordering dependencies, and path normalization. If it's an HTTP client, think about redirect credential stripping, encoding detection, and connection state leaking. If it's a serialization library, think about null handling asymmetry, API surface consistency between direct methods and view wrappers, lazy evaluation caching bugs, and round-trip fidelity. If it's a web framework, think about response helper edge cases, configuration compilation chains, and middleware state isolation. If it's a batch processor, think about crash recovery, idempotency, silent data loss, state corruption. If it handles randomness or statistics, think about seeding, correlation, distribution bias.
- "What produces correct-looking output that is actually wrong?" — This is the most dangerous class of bug: output that passes all checks but is subtly corrupted. A response with a
200 OK but the wrong Content-Type. A redirect that succeeds but leaks credentials. A deserialized object that has silently truncated values.
- "What happens at 10x scale that doesn't happen at 1x?" — Chunk boundaries, rate limits, timeout cascading, memory pressure.
- "What happens when this process is killed at the worst possible moment?" — Mid-write, mid-transaction, mid-batch-submission.
- "Where do two surfaces that should behave the same drift on edge inputs?" — Overloads, aliases, sync/async APIs, builder vs direct APIs, direct mutators vs live views/wrappers, stdlib-compatible wrappers vs framework-native surfaces. For Java/Kotlin:
add(null) vs asList().add(null), put(key,null) vs asMap().put(key,null). For Python: constructor encoding vs mutator encoding, sync vs async client behavior.
- "What emits plausible output with subtly wrong metadata?" — Content type, charset, route pattern, ETag strength, byte count, auth/header/cookie propagation, status code, cache validators.
- "What standard grammar or list syntax is being parsed with ad hoc string logic?" — Quality values (
q=0), comma-separated headers, digest challenges, MIME types with parameters, query strings, enum/keyword sets, cookie merging.
- "What edge-case inputs would a domain expert reach for?" — For HTTP code:
Accept-Encoding: gzip;q=0, Connection: keep-alive, Upgrade, Content-Type: application/problem+json. For serialization code: null through different API surfaces, values at Integer.MAX_VALUE + 1, round-tripping through encode-then-decode. For routing code: overlapping patterns, mounted prefix propagation, same path with different methods.
- "What information does the user need before committing to an irreversible or expensive operation?" — Pre-run cost estimates, confirmation of scope (especially when fan-out or expansion will multiply the work), resource warnings. If the system can silently commit the user to hours of processing or significant cost without showing them what they're about to do, that's a missing safeguard. Search for operations that start long-running processes, submit batch jobs, or trigger expansion/fan-out — and check whether the user sees a preview, estimate, or confirmation with real numbers before the point of no return.
- "What happens when a long-running process finishes — does it actually stop?" — Polling loops, watchers, background threads, and daemon processes that run until completion should have explicit termination conditions. If the loop checks "is there more work?" but never checks "is all work done?", it will run forever after completion. This is especially common in batch processors and queue consumers.
Generate at least 5 ranked failure scenarios from this knowledge. You don't need to have observed these failures — you know from training that they happen to systems of this type. Write them as specific bug hypotheses with file-path and line-number citations, ranked by priority. Frame each as: "Because [code at file:line] does [X], a [domain-specific edge case] will produce [wrong behavior] instead of [correct behavior]." Then ground them in the actual code you explored: "Read persistence.py line ~340 (save_state): verify temp file + rename pattern."
Anti-patterns that fail the gate: A Quality Risks section that lists defensive patterns the code already has (things the code does right) is not a risk analysis — it is a reassurance exercise and will not find bugs. A section that lists risky modules without specific failure scenarios is not actionable. A section that concludes "this is a mature, well-tested library so basic bugs are unlikely" is actively harmful — mature libraries have the most subtle API-contract and edge-case bugs, precisely because the obvious ones were found years ago. The test: could a code reviewer read each scenario and immediately know what function to open and what input to test? If not, the scenario is too abstract.
Step 7: Derive Testable Requirements
Read references/requirements_pipeline.md for the complete five-phase pipeline, domain checklist, and versioning protocol.
This is the most important step for the code review protocol. Everything found during exploration — specs, ChangeLog entries, config structs, source comments, chat history — gets distilled into a set of testable requirements that the code review will verify. The pipeline separates contract discovery from requirement derivation, uses file-based external memory, and includes mechanical verification with a completeness gate.
Why this matters: Structural code review catches about 65% of real defects. The remaining 35% are intent violations — absence bugs, cross-file contradictions, and design gaps. These are invisible to code reading because the code that IS there is correct. You need to know what the code is supposed to do, then check whether it does it. That's what testable requirements provide.
The five-phase pipeline:
-
Phase A — Contract extraction. Read all source files, list every behavioral contract. Write to quality/CONTRACTS.md. This is discovery — list everything, even if it seems obvious.
-
Phase B — Requirement derivation. Read CONTRACTS.md and documentation. Group related contracts, enrich with user intent, write formal requirements. Write REQ records to quality/requirements_manifest.json (source of truth) and render to quality/REQUIREMENTS.md. For each requirement, record the tier (1–5 per schemas.md §3.1) and — when tier ∈ {1, 2} — the citation block produced by bin/reference_docs_ingest invoking bin/citation_verifier per schemas.md §5.4 / §5.5. The LLM does not shell out to citation_verifier directly; the excerpt is a product of the ingest pipeline and is re-verified by quality_gate.py at gate time. For Tier 3 REQs (code-is-the-spec), cite the source file:line in the description; citations are for FORMAL_DOC references only and must not appear on Tier 3/4/5 REQs. The tier + citation pair creates the forward link in the traceability chain: reference_docs/cite → requirements → bugs → tests. See the tier/citation framing block later in this step for the full field list and the Tier-1-wins-over-Tier-2 rule.
Optional Pattern: field on REQs. A requirement that needs a Phase 3
compensation grid should declare its pattern class:
Pattern: whitelist — authoritative list of items, every site must handle
each one.Pattern: parity — symmetric operations that must match
(encode↔decode, setup↔teardown).Pattern: compensation — sites that must compensate for a shared gap.
Missing the field means no grid. Setting an invalid value fails
quality_gate.py.
Preservation rule (Phase 2). While Pattern: is optional in the design
sense (some REQs are single-site and need no grid), it is REQUIRED to
preserve when the Phase-1 hypothesis already carried it. Phase 2 must
transcribe Pattern: from EXPLORATION.md to quality/REQUIREMENTS.md and
quality/requirements_manifest.json whenever present. Silent omission is a
documented v1.4.5-regression vector — the Phase 5 cardinality gate cannot
enforce coverage on a REQ it doesn't know is pattern-tagged. The gate's
structural backstop (C13.7/Fix 2) cross-checks REQs that carry per-site UC
references (UC-N.a/UC-N.b form emitted by Phase 1's Cartesian UC rule)
and fails the gate if Pattern is missing on such a REQ.
Primary-source extraction rule for code-presence claims. When writing a requirement that asserts specific constants, values, or labels are handled by a specific function (e.g., "the whitelist must preserve X, Y, and Z"), the requirement must distinguish between what the spec says should be there and what the code actually contains. Extract the actual contents from the code (case labels, map keys, if-else branches) and compare to the spec's list. If a constant appears in the spec but NOT in the code, write the requirement as "must handle X — [NOT IN CODE]: defined in header.h:NN but absent from function() at file.c:NN-NN." Do not write "must preserve X" without verifying X is actually preserved. This prevents a contamination chain where a requirement asserts code presence, the code review copies the assertion, the spec audit inherits it, and the triage accepts it — all without anyone reading the actual code. This exact chain was observed in v1.3.17 virtio testing: REQUIREMENTS.md asserted RING_RESET was preserved in a switch, the code review copied the list, three spec auditors inherited the claim, and the bug went undetected.
Mechanical verification artifact for dispatch functions (mandatory). When a contract asserts that a function handles, preserves, or dispatches a set of named constants (feature bits, enum values, opcode tables, event types, handler registries), you must generate and execute a shell command or script that mechanically extracts the actual case labels/branches from the function body before writing the contract line. Save the raw output to quality/mechanical/<function>_cases.txt. The command must be a non-interactive pipeline (e.g., awk + grep) that cannot hallucinate — it reads file bytes and prints matches. Example:
awk '/void vring_transport_features/,/^}$/' drivers/virtio/virtio_ring.c \
| grep -E '^\s*case\s+' > quality/mechanical/vring_transport_features_cases.txt
After execution, read the output file and use it as the sole source of truth for what the function handles. A contract line asserting "function preserves constant X" is forbidden unless quality/mechanical/<function>_cases.txt contains a matching case X: line. If a constant appears in a spec or header but NOT in the mechanical output, the contract must record it as absent: "must handle X — **[NOT IN CODE]**: defined in header.h:NN but absent from function() per mechanical check." Downstream artifacts (REQUIREMENTS.md, RUN_SPEC_AUDIT.md, code review) must cite the mechanical file path when referencing dispatch-function coverage — they may not replace the mechanical output with a hand-written list.
Mechanical artifact integrity check (mandatory). For each mechanical extraction command, also append it to quality/mechanical/verify.sh as a verification step. The script must re-run the same extraction pipeline and diff the result against the saved file. Generate verify.sh with this structure:
#!/bin/bash
set -euo pipefail
FAIL=0
ACTUAL=$(awk '/void vring_transport_features/,/^}$/' drivers/virtio/virtio_ring.c | grep -nE '^\s*case\s+')
SAVED=$(cat quality/mechanical/vring_transport_features_cases.txt)
if [ "$ACTUAL" != "$SAVED" ]; then
echo "MISMATCH: vring_transport_features_cases.txt"
diff <(echo "$ACTUAL") <(echo "$SAVED") || true
FAIL=1
else
echo "OK: vring_transport_features_cases.txt"
fi
exit $FAIL
Phase 6 must execute bash quality/mechanical/verify.sh and the benchmark fails if any artifact mismatches. This catches a failure mode observed in v1.3.19: the model executed the extraction command but wrote its own expected output to the file instead of letting the shell redirect capture it, inserting a hallucinated case VIRTIO_F_RING_RESET: line that the real command does not produce. Re-running the same command in a separate step and diffing against the file detects this tampering.
Immediate integrity gate (mandatory, Phase 2a). Run bash quality/mechanical/verify.sh immediately after writing each *_cases.txt file and before writing any contract, requirement, or prose artifact that cites the extraction. If exit code ≠ 0: stop, delete the failed *_cases.txt, re-run the extraction command with a fresh shell redirect (do not hand-edit the output), and re-verify. Do not advance to Phase 3/2c until verify.sh exits 0. Save verify.sh stdout and exit code to quality/results/mechanical-verify.log and quality/results/mechanical-verify.exit as durable receipt files. This gate exists because v1.3.23 showed that deferring verification to Phase 6 allows downstream artifacts (CONTRACTS.md, REQUIREMENTS.md, triage probes) to build on a forged extraction — the model reconciles a discrepancy between requirements and the artifact by corrupting the artifact instead of correcting the requirement.
Mechanical artifacts are immutable after extraction. Once a *_cases.txt file has been written by the shell redirect and verified by verify.sh, it must not be modified, overwritten, or regenerated for the remainder of the run. If a downstream step discovers a discrepancy between the mechanical artifact and a requirement or contract, the requirement or contract is wrong — not the artifact. Fix the prose, not the extraction. This rule prevents the v1.3.23 failure mode where the model overwrote a correct extraction with fabricated content to match its own narrative.
Forbidden probe pattern (triage and verification). Triage probes, verification probes, and audit assertions must not use open('quality/mechanical/...') or cat quality/mechanical/... as sole evidence for what a source file contains at a given line. To verify that function F handles constant C at line N, the probe must either: (a) read the source file directly (open('drivers/virtio/virtio_ring.c') with line-anchored assertions), or (b) re-execute the same extraction pipeline used by verify.sh and check its output. Reading the saved artifact proves only what the artifact says, not what the code says — this is circular verification. In v1.3.23, Probe C validated the forged artifact instead of the source code, passing with fabricated data.
Do not create an empty mechanical/ directory. Only create quality/mechanical/ if the project's contracts include dispatch functions, registries, or enumeration checks that require mechanical extraction. If no such contracts exist, skip the directory entirely and record in PROGRESS.md: Mechanical verification: NOT APPLICABLE — no dispatch/registry/enumeration contracts in scope. Creating an empty mechanical/ directory (or one without verify.sh) is non-conformant — it signals that extraction was attempted and abandoned. Decide before creating the directory: does this project have dispatch-function contracts? If no, don't mkdir. If yes, populate it fully.
Normative vs. descriptive split. Requirements and contracts must use normative language ("must preserve," "should handle") for expected behavior. They may only use descriptive language ("preserves," "handles") when the mechanical verification artifact confirms the claim. A requirement that says "the implementation preserves VIRTIO_F_RING_RESET" without a confirming mechanical artifact is non-conformant — write "the implementation must preserve VIRTIO_F_RING_RESET" and cite the mechanical check result showing whether the constant is currently present or absent.
-
Phase C — Coverage verification. Cross-reference every contract against every requirement. Fix gaps. Loop up to 3 times until coverage reaches 100%. Write to quality/COVERAGE_MATRIX.md. The matrix must have one row per requirement (REQ-001, REQ-002, etc.) — not grouped ranges like "C-001 to C-007 | REQ-001, REQ-003". Grouped ranges make machine verification impossible and hide gaps.
-
Phase D — Completeness check + self-refinement loop. Apply the domain checklist, testability audit, and cross-requirement consistency check. Also verify that every deep document with a "will cover" commitment in the coverage commitment table has at least one requirement traced to it — if not, add requirements for the gap before continuing.
Write to quality/COMPLETENESS_REPORT.md as a baseline completeness report (without a ## Verdict section — the verdict is deferred to Phase 5 post-reconciliation, which produces the only verdict that counts for closure). Then run up to 3 self-refinement iterations: read the report, fix gaps, re-check. Short-circuit when fewer than 3 changes per iteration.
-
Phase E — Narrative pass. Add project overview (with overview validation gate), then derive use cases (with use case derivation gate). Both gates must pass before proceeding to category narratives, cross-cutting concerns, and final reordering. This sequencing prevents multi-pass loops where a failed late gate forces re-derivation. Reorder for top-down flow. Renumber sequentially.
REQUIREMENTS.md must begin with a human-readable overview that answers: What is this project? What does it do? Who are the actors (users, systems, hardware, protocols)? What are the highest-risk areas? This overview should be useful to someone who has never seen the project before. If the project is a library or driver where all actors are systems, describe the system actors (kernel maintainers, protocol peers, integrators, end-user developers) and their interactions. Do not start with raw scope metadata or HTML comments — lead with a plain-language description.
Overview validation gate (mandatory). After writing the overview, perform this self-check before proceeding to use case derivation:
Does this overview describe the project the way its actual users would recognize it? Specifically:
- Does it name the project's ecosystem role and real-world significance?
- Does it identify who depends on it and for what?
- Would a developer who uses this project daily say "yes, that's what it is and why it matters"?
- For well-known projects, does it reflect publicly known adoption (e.g., Cobra → kubectl/Hugo/GitHub CLI; Express → millions of Node.js API servers; Zod → form validation/tRPC; Serde → the default Rust serialization layer)?
If the overview reads like it was written by someone who only read the source code and never used the software, revise it before proceeding. The overview sets the frame for everything downstream — feature-oriented use cases and internally focused requirements are symptoms of an overview that only describes the code, not the project.
Use case derivation (mandatory, runs after overview gate). Derive 5–7 use cases from the validated overview and gathered documentation, then validate them against the code. Each use case must:
- Describe a real user outcome, not a code feature. "Developer builds a CLI tool with nested subcommands, persistent flags, and shell completion" — not "Framework supports command trees."
- Name a concrete actor and what they are trying to accomplish. Actors include end-user developers, system administrators, kernel maintainers, protocol peers, integrators, and automated consumers.
- Be recognizable to an actual user of the software. For well-known projects, validate use cases against the model's own knowledge of the project, community docs, tutorials, and real-world adoption patterns.
- Connect to at least one requirement through testable conditions of satisfaction.
The pipeline should explicitly ask: "Based on this project's overview, gathered documentation, and known user base, what are the 5–7 most important things real users do with this software?" Derive use cases from that question — not from scanning the code and grouping features into categories.
