| name | quality-playbook |
| description | Run a complete quality engineering audit on any codebase. Derives behavioral requirements from the code, generates spec-traced functional tests, runs a three-pass code review with regression tests, executes a multi-model spec audit (Council of Three), and produces a consolidated bug report with TDD-verified patches. Finds the 35% of real defects that structural code review alone cannot catch. Works with any language. Trigger on 'quality playbook', 'spec audit', 'Council of Three', 'fitness-to-purpose', or 'coverage theater'. |
| license | Complete terms in LICENSE.txt |
| metadata | {"version":"1.5.8","author":"Andrew Stellman","github":"https://github.com/andrewstellman/quality-playbook"} |
Quality Playbook Generator
Plan Overview — read this first, then explain it to the user
Before reading any other section of this skill, understand the plan and its dependencies. Each phase produces artifacts that the next phase depends on. Skipping or rushing a phase means every downstream phase works from incomplete information.
Phase 0 (Prior Run Analysis): If previous quality runs exist, load their findings as seed data. This is automatic and only applies to re-runs.
Phase 1 (Explore): Run the documentation intake first. Invocation form: bin/reference_docs_ingest.py does from bin import benchmark_lib at import, so Python's -m bin.reference_docs_ingest resolves ONLY when bin/ is reachable as a top-level package. Resolve the install root (the directory containing this SKILL.md) via the same install-location fallback list this skill uses for references/ / quality_gate.py (the 10 canonical layouts documented below — install-root forms: <empty>, .claude/skills/quality-playbook, .github/skills, .github/skills/quality-playbook, .cursor/skills/quality-playbook, .continue/skills/quality-playbook, .codex/skills/quality-playbook, .windsurf/skills/quality-playbook, .cline/skills/quality-playbook, .aider/skills/quality-playbook). Then, from the target repo root, run PYTHONPATH=<install_root> python3 -m bin.reference_docs_ingest <target>. If <install_root> is empty (QPB self-bootstrap, or a setup_repos.sh install where bin/ is already at the target root) omit the PYTHONPATH= prefix — cwd already provides bin/. The bare python -m bin.reference_docs_ingest <target> works ONLY for those two layouts; an install_skill.py-layout adopter (bin/ inside the marker dir) otherwise gets ModuleNotFoundError: No module named 'bin'. This walks reference_docs/ — cite/ files produce quality/formal_docs_manifest.json records; top-level files are loaded as Tier 4 context via reference_docs_ingest.load_tier4_context(<target>). Then explore the codebase in three stages: open exploration driven by domain knowledge, domain-knowledge risk analysis, and selected structured exploration patterns. Write all findings to quality/EXPLORATION.md. This file is the foundation — Phase 2 reads it as its primary input.
Phase 2 (Generate): Read EXPLORATION.md and produce the quality artifacts: requirements, constitution, functional tests, code review protocol, integration tests, spec audit protocol, TDD protocol. (AGENTS.md at the target's repo root is generated by the orchestrator AFTER Phase 6, not by you in Phase 2 — see "File 6" below for the contract.)
Phase 3 (Code Review): Run the three-pass code review against HEAD. Write regression tests for every confirmed bug. Generate patches.
Phase 4 (Spec Audit): Three independent AI auditors review the code against requirements. Triage with verification probes. After triage, the same Council runs the Layer-2 semantic citation check — one prompt per reviewer, structured per-REQ verdicts for every Tier 1/2 citation, output to quality/citation_semantic_check.json. Write regression tests for net-new findings.
Phase 5 (Reconciliation): Close the loop — every bug from code review and spec audit is tracked, regression-tested or explicitly exempted. Run TDD red-green cycle. Finalize the completeness report.
Phase 6 (Verify): Run self-check benchmarks against all generated artifacts. Check for internal consistency, version stamp correctness, and convergence.
Phase 7 (Present, Explore, Improve): Present results to the user with a scannable summary table, offer drill-down on any artifact, and provide a menu of improvement paths (iteration strategies, requirement refinement, integration test tuning). This is the interactive phase where the user takes ownership of the quality system.
Every bug found traces back to a requirement, and every requirement traces back to an exploration finding.
The critical dependency chain: Exploration findings → EXPLORATION.md → Requirements → Code review + Spec audit → Bug discovery. A shallow exploration produces abstract requirements. Abstract requirements miss bugs. The exploration phase is where bugs are won or lost.
MANDATORY FIRST ACTION: After reading and understanding the plan above, print the following attribution banner exactly as written below — do NOT condense, abbreviate, summarize, reformat, or add a version number. Reproduce the full 8-line canonical block including the two ===…=== rules, both taglines, the blank lines, and the license line, byte-for-byte matching bin/_purpose.print_attribution_banner() (the single source of truth). Then explain the plan in your own words — what you'll do, what each phase produces, and why the exploration phase matters most. Emphasize that exploration starts with open-ended domain-driven investigation, followed by domain-knowledge risk analysis that reasons about what goes wrong in systems like this, then supplemented by selected structured patterns. Do not copy the plan verbatim; paraphrase it to demonstrate understanding.
================================================================================
Quality Playbook -- by Andrew Stellman
https://github.com/andrewstellman/quality-playbook
AI code review is good. Quality engineering is better.
Because code that looks right can still do the wrong thing.
Licensed under the Apache License, Version 2.0
================================================================================
Generate a complete quality system tailored to a specific codebase. Unlike test stub generators that work mechanically from source code, this skill explores the project first — understanding its domain, architecture, specifications, and failure history — then produces a quality playbook grounded in what it finds.
How to run this — self-encoded invocation contract
If the operator hands you this skill (or points you at any QPB-installed target) and says "Run the Quality Playbook" — possibly with a hint like "this is a bootstrap run" or "run on itself" or "self-audit" — this section tells you exactly what to do. The operator should not need to provide additional instructions; the canonical invocation, the defaults, the guardrails, and the output contract all live here.
Pick your execution mode
QPB ships in two execution shapes. Pick the one that matches your runtime — the wrong choice produces the codex-on-codex indirection pathology.
| Mode | When this is you | What you do |
|---|
| A. Skill-direct (UI-context) | You are a coding agent (Claude Code, Cursor, Copilot, Codex desktop, etc.) handed this skill in your own chat. Your runtime IS the reasoning loop — you read files, you write files, you decide. | Walk through Phase 1 → Phase 6 yourself using the externalized phase prompts in phase_prompts/. Write artifacts into the target's quality/ directory directly. No subprocess, no runner. |
| B. Runner-driven (CLI-automation) | The operator is invoking python3 -m bin.run_playbook deliberately — to batch across multiple targets, drive a headless CI run, or fan out per-phase work to a different model than the one reading this prose. | The orchestrator spawns a CLI agent (claude, copilot, codex, or cursor) per phase. You (or whoever is reading this) are the operator-side control loop, not the per-phase reasoner. |
Both modes use the same phase prompt content — the phase_prompts/*.md files at the repo root are the single source of truth, loaded by bin/run_playbook.py::_load_phase_prompt and read directly by Mode A walkthroughs. The only thing the two modes differ on is WHO drives — you (Mode A) or the orchestrator subprocess-spawning a CLI agent (Mode B).
When in doubt, default to Mode A. If the operator wanted runner-driven invocation they would have run the runner themselves; if they pasted "Run the Quality Playbook" into your chat, they want you to drive. The Mode B section below tells you what to do if the operator explicitly invokes the runner.
Documented Mode A vs Mode B asymmetries — by design. Three behaviors deliberately differ between the modes because the modes serve different purposes (Mode A is the adopter-facing walkthrough on a target repo; Mode B is the QPB-internal runner used for benchmarking + headless CI). The phase prompts are identical (single source of truth — see paragraph above); the divergences are in the driver obligations, not the audited contract:
- Phase 0 entry contract — install validator (Mode A only). Mode A's mandatory first action is to run the install validator. Invocation form is install-location-aware: resolve the install root (the directory containing this SKILL.md) via the same install-location fallback list this skill uses for
references/ / phase_prompts/ (the 10 canonical layouts documented below), then run python3 <install_root>/bin/qpb_validate.py <target>. For a clone / self-bootstrap (install root is the QPB clone, bin/ reachable from cwd) the bare python3 bin/qpb_validate.py <target> works. For an install_skill-layout adopter (channel or manual install — bin/ inside the marker dir, e.g. .github/skills/quality-playbook/bin/) you MUST use the install-root-prefixed form; running bare bin/qpb_validate.py from the target repo root fails (the validator lives under the marker dir, not at repo root). The adopter has just installed QPB into their target repo and must confirm canonical layout before any phase. Mode B (the QPB clone running its own runner) operates from the canonical install by construction; the validator would always succeed against the QPB clone itself, so the check would be vacuous. Adopters running Mode A get the install correctness gate; Mode B operators have it already.
- End-of-run archive (Mode B only). Mode B's
_finalize_iteration calls archive_lib.archive_run after Phase 6, materializing quality/previous_runs/<ts>/ for benchmark replay + cross-run diffing. Mode A produces artifacts directly into the target's live quality/ for the adopter to inspect, commit, or re-run on top of; there is no archive partition because there is no benchmark loop. Adopters who want longitudinal archives invoke Mode B explicitly (python3 -m bin.run_playbook --full-run <target>).
- Phase 6 auditor-prompt sub-agent dispatch (Mode A only).
phase_prompts/phase6.md routes Mode A through phase_prompts/phase6_auditor.md via a fresh-context sub-agent spawn (the A-13 hybrid). Mode B's per-phase CLI subprocess is already a fresh context (the structural separation the auditor exists to provide), so phase6.md:5-7 tells Mode B to execute the verification inline and skip the auditor spawn. phase_prompts/phase6_auditor.md is therefore only loaded under Mode A; the file's header documents this routing explicitly.
These asymmetries are pinned by bin/tests/test_mode_a_b_parity_documented.py. If a future change wants to align them, update the pin test, the rationale here, and add the missing behavior to the appropriate side.
Mode A — skill-direct walkthrough (UI-context)
The operator's prompt is just "Run the Quality Playbook" (or "run on itself", "self-audit", etc.). You drive every phase inline.
Mode A means YOU execute the skill. If you are an interactive coding agent (Claude Code, Cursor, Copilot UI, Codex desktop, any tool where the operator is monitoring your chat output in real-time), DO NOT:
- Spawn a sub-skill via your
quality-playbook skill-invocation mechanism.
- Spawn a sub-agent via your Task tool / Agent tool / similar.
- Invoke
python3 -m bin.run_playbook — that's Mode B, the CLI-automation path.
- Read
agents/quality-playbook.agent.md or agents/quality-playbook-claude.agent.md and follow their "spawn sub-agents per phase" pattern — those files are AUTOMATION ONLY (batch CI runs, headless invocations) where per-phase context-window isolation is necessary at the cost of operator chat visibility.
Phase 0 entry contract (MANDATORY first action): run the QPB install validator. Resolve the install root via the install-location fallback list (per the Phase 0 numbered step above), then run python3 <install_root>/bin/qpb_validate.py <target-repo> — bare python3 bin/qpb_validate.py <target-repo> works only when cwd already contains bin/ (QPB clone / self-bootstrap); install_skill-layout adopters (channel install — bin/ under .github/skills/quality-playbook/, etc.) MUST prefix the install root. Paste every event= line; do not proceed until event=validation_complete status=ok — see AGENTS.md "Mode A entry sequence". Non-negotiable — skipped/fabricated validation and bare-path-from-repo-root invocation are both documented adopter failure modes.
EXCEPTION: Phase 6 verification MUST be delegated to a fresh-context sub-agent. Sub-agent delegation is forbidden for EXECUTION (Phases 1-5) because hiding work from operator chat defeats real-time monitoring. Sub-agent delegation is MANDATED for VERIFICATION (Phase 6) because shared-context executor-as-verifier has empirically fabricated PASS verdicts against failing gates across multiple benchmark runs despite repeated rounds of strengthened prompt-level witness language. The principle is the same as the rule above — operator chat should carry the truth — but the mechanism is opposite: visibility for execution; isolation for verification (a fresh auditor has none of the executor's memory/anchoring/completion-reward bias). See phase_prompts/phase6.md + phase_prompts/phase6_auditor.md for the sub-agent delegation protocol. This exception is scoped strictly to Phase 6 verification — the no-sub-agent rule still holds for all Phase 1-5 execution.
The failure mode that motivated this rule: a shared-context session ran phases 1-6, fabricated the gate-verdict log (quality/results/quality-gate.log was hand-written with a "PASS" tally while the actual gate showed 14 FAIL, GATE FAILED), and reported PASS to the parent. The Phase 6 witness contract (your obligation to quote the gate's verdict lines verbatim) cannot enforce when a sub-skill owns the witness — only the parent's chat carries operator-readable evidence.
YOU read SKILL.md and the phase prompts. YOU execute Phases 1-5 yourself and write artifacts to quality/. Phase 6 verification is delegated to the fresh-context auditor sub-agent per the exception above (the auditor invokes quality_gate.py); you paste its verbatim verdict. Your chat IS the trust trail — execution stays visible, verification stays bias-isolated.
You have shell access — probe before claiming unavailability. Mode A means an interactive coding agent (Claude Code, Cursor, Copilot UI) running commands on the operator's actual machine via your Bash tool — NOT a sandboxed runtime. A common failure is writing "the sandbox can't run X" reflexively, conflating your model-level "I can't natively execute" with the harness Bash tool you DO have. A which <tool> / <tool> --version probe costs one line; assuming-and-being-wrong forfeits the entire Phase 5 evidence trail.
For each phase 1..6, in order:
- Load the phase prompt. Read
phase_prompts/phaseN.md (resolve via the same install-location fallback list documented for references/ below). For phase1.md, substitute {seed_instruction} (the prelude that says "skip Phase 0/0b" — empty string when seeds are allowed) and {role_taxonomy} (the taxonomy block rendered from the role taxonomy below). For phase2.md through phase6.md, the file is pure-literal — read it verbatim.
- Execute the phase per the prompt. Read the inputs the prompt names, do the analysis, write outputs into the target's
quality/ directory.
- STOP at the end-of-phase boundary. Every phase prompt ends with an "IMPORTANT: Do NOT proceed to Phase N+1" instruction. Honor it. The operator advances to the next phase by saying so.
You are responsible — without the orchestrator's structural backstop — for the same source-unchanged invariant the runner enforces: do NOT modify any file outside the target's quality/ directory. In Mode B the gate would catch this; in Mode A you are the gate. A documented failure mode: a Phase 2 LLM modifying the target's root AGENTS.md — the same failure mode applies in Mode A.
For the bootstrap-run (self-audit) variant of Mode A, see "Bootstrap mode" below — the only delta is that the target IS the QPB repo, so cite the same phase_prompts/ files you read from.
Mode A scope — what's covered, what's Mode-B-only
The per-phase walkthrough above scopes Mode A to phases 1..6. The following surfaces are deliberately Mode-B-only — if the operator wants them, point them at the runner instead of trying to drive them yourself:
- Phase 0 / Phase 0b (seed injection from prior runs). The orchestrator handles seed discovery, prior-run scanning, and seed-prompt injection. In Mode A, treat every run as
--no-seeds (skip Phase 0/0b entirely, start at Phase 1). If the operator explicitly asks for seed-driven exploration, hand off to Mode B (python3 -m bin.run_playbook --with-seeds <target>).
- Phase 7 (interactive Present / Explore / Improve). Mode A's Phase 7 is the conversational version — present the artifact summary table inline and let the operator drive what to explore. See the Phase 7: Present, Explore, Improve section below for the canonical treatment covering both modes.
- Iteration strategies (gap / unfiltered / parity / adversarial). Iterations re-enter the playbook with a strategy-specific addendum. In Mode A, after Phase 6 completes cleanly, hand off to Mode B for iterations:
python3 -m bin.run_playbook --next-iteration --strategy <name> <target>. The iteration prompts (phase_prompts/iteration.md) ARE single-source-of-truth, but the iteration-orchestration loop (rotating through gap → unfiltered → parity → adversarial) is the runner's job. A Mode A operator who wants iterations after Phase 6 should be told: "Phase 6 is done; run python3 -m bin.run_playbook --full-run <target> to get all four iteration strategies, or pick one strategy explicitly with --next-iteration --strategy gap."
If the operator asks for one of these surfaces in Mode A and the request is ambiguous (e.g., "also do the iterations"), surface the mode-handoff explicitly rather than improvising — improvisation is how the prompt content drifts away from the runner's canonical loop.
Mode B — runner-driven invocation (CLI-automation)
The operator runs python3 -m bin.run_playbook themselves (typically because they want batching, headless CI, or to route per-phase work to a different model). The orchestrator at bin/run_playbook.py spawns a CLI agent per phase, feeds it the externalized phase prompt, and aggregates the result.
Canonical invocation
The orchestrator is the entry point. Two invocation forms are supported:
python3 -m bin.run_playbook <target> — canonical package-module form.python3 /path/to/QPB/bin/run_playbook.py <target> — direct script form. Works from any cwd because the module injects QPB root into sys.path before importing sibling modules.
<target> is the path to the project to audit. For a bootstrap run (target IS the QPB repo), pass . from the repo root. For any other target, pass the path to that target's repo root.
Default behavior (no flags)
Bare invocation triggers a full run: all 6 phases (Explore → Generate → Code Review → Spec Audit → Reconciliation → Verify) followed by all 4 iteration strategies (gap → unfiltered → parity → adversarial), executed synchronously in the same session. Any prior quality/ directory is auto-archived to quality/previous_runs/<TIMESTAMP>/ before the new run starts.
This is the canonical operator path. Don't ask permission to add flags; the defaults are the answer.
When the bare invocation fires, the orchestrator emits a one-line stderr banner naming the run cost. That banner is informational; let it scroll.
Common overrides
Use only when the operator asks for something specific:
| Need | Flag | Effect |
|---|
| Run a single phase | --phase N (where N ∈ 1..6) | Use --phase 1 for an explore-only run. |
| Skip iteration strategies | omit --iterations and pass --phase 1,2,3,4,5,6 | Phases run; iterations don't. |
| Specific iteration | --strategy <name> --next-iteration | Iterates on an existing quality/ run with a chosen strategy. |
| Multi-target | pass several positional targets | Each runs independently. |
| Per-phase CLI agent | --claude / --copilot / --codex / --cursor | Picks which CLI runner the orchestrator spawns. Default is --copilot. The --cursor runner requires cursor-cli 3.1+. |
Recovering from a partial / aborted runner-driven run
Council 2026-04-30 P1-4: the operator-hygiene guidance for cleaning up after an aborted run lives in the Bootstrap mode section below ("Bootstrap-run operator hygiene") — the recovery is identical in Mode B: git restore quality/ to discard the partial Phase 1/2 output, then re-invoke. Do NOT edit files outside quality/ to "tidy up" — the source-unchanged invariant trips on the very next run. See the Bootstrap-mode hygiene paragraph for the full mechanic; it applies regardless of whether the abort happened during a self-audit run or against an external target.
Bootstrap mode (running QPB on itself)
When the operator says "this is a bootstrap run" or "we're running QPB on itself" or "self-audit":
- Confirm the working directory is the QPB repo root (or
cd there).
- Invoke
python3 -m bin.run_playbook . — same canonical form, target is ..
- The orchestrator handles archival of the existing
quality/ tree to quality/previous_runs/<TIMESTAMP>/ automatically; you don't need to clean anything manually.
The run proceeds the same way as any other target. The only difference is that the audit subject IS the playbook itself, so the produced artifacts describe QPB's own quality system.
Bootstrap-run operator hygiene — recovering from a partial / aborted run. If a prior bootstrap run aborted mid-flight (e.g., the source-unchanged invariant tripped, a phase prompt errored, the operator hit Ctrl-C), the working tree may contain a half-written quality/ directory plus a quality/previous_runs/<TIMESTAMP>/.partial sentinel marking the abandoned archive. Before re-invoking, run git restore quality/ (and, if you want a clean slate, git clean -fd quality/) to drop any uncommitted Phase 1/2 output from the aborted run. The orchestrator will re-archive the now-pristine quality/ tree and start clean. Do NOT edit files outside quality/ to "tidy up" — anything outside quality/ is QPB source; touching it for cleanup will trip the source-unchanged invariant on the very next run. The 2026-04-30 bootstrap test surfaced this exact recovery question: the operator had a half-written quality/ from an aborted Phase 2 and re-running without restoring left stale Phase 1 artifacts that confused the next run's archival.
Guardrails (machine-checkable; treat as hard constraints)
These are not suggestions; the orchestrator enforces them and a violation aborts the run:
- Synchronous execution — no sub-agent delegation. Run every phase yourself in the same session. Do NOT use the Task tool, sub-agent dispatch, background-agent invocations, the
quality-playbook skill-invocation mechanism, or any "delegate phases 2–6 to a worker" pattern. Two verified failure modes the guardrail closes: (a) Phase 1 completes, phases 2–6 silently die in a delegated agent that loses its parent session, the runner self-marks -PARTIAL, and the operator gets no signal anything was wrong. (b) An interactive coding session spawns the quality-playbook orchestrator sub-skill; sub-skill runs Mode A in its own context but fabricates quality/results/quality-gate.log and reports "Gate: PASS" against an actual failing gate. The Phase 6 witness contract cannot enforce when the witness lives in a sub-skill's chat that the parent never sees. Phase 6 verification — mandatory sub-agent delegation: the no-sub-agent rule above governs Phase 1-5 EXECUTION (operator visibility). Phase 6 VERIFICATION is the opposite: it MUST run in a fresh-context sub-agent, because shared-context executor-as-verifier has empirically fabricated PASS verdicts against failing gates despite repeated rounds of strengthened witness language. Same principle (operator chat carries the truth), opposite mechanism (visibility for execution; bias-isolation for verification). Scoped strictly to Phase 6 verification — see phase_prompts/phase6.md + phase_prompts/phase6_auditor.md.
- Don't patch QPB source mid-run. If you encounter a bug in
bin/, .github/skills/, agents/, references/, SKILL.md, or AGENTS.md during the run, STOP and report: name the file:line, describe the failure, propose a fix shape — but do NOT apply the fix. The orchestrator captures a git-SHA baseline at run start and verifies the source tree unchanged at every phase boundary; an autonomous patch fails the gate with a diagnostic naming the modified files. Patches go through Council review, not mid-run improvisation.
- Don't delete sentinel files. Files protected by
.gitignore !-rules (e.g., reference_docs/.gitkeep, reference_docs/cite/.gitkeep) keep otherwise-empty tracked directories present. The pre-flight check enumerates every !-rule and aborts if any sentinel is missing. If you find such a file and don't understand its purpose, leave it alone.
- Phase 1 file enumeration uses
git ls-files. Use git ls-files as the canonical file list when the target is a git repo; this respects .gitignore automatically. Do NOT use os.walk, find, os.listdir, or any recursive directory walker — those pull in .git/, .venv/, node_modules/, build outputs, and vendored dependencies, all of which the role-map validator rejects. Disallowed path prefixes are .git/, .venv/, venv/, node_modules/, __pycache__/, .pytest_cache/, .mypy_cache/, .ruff_cache/, .tox/, plus any path whose components end in .egg-info or .dist-info. The role map carries a provenance field recording which enumeration source you used ("git-ls-files" or "filesystem-walk-with-skips" for non-git targets). There is also a 2000-entry ceiling; a role map exceeding it almost certainly walked .gitignored content.
- Cross-artifact agreement. EXPLORATION.md's "File inventory" section and the role map's
summary field both render from bin.role_map.summarize_role_map(). Don't write file counts or role percentages by hand; copy from the helper. The validator cross-checks the two and rejects mismatches.
If the operator's prompt says something that conflicts with these guardrails (e.g., "delegate phases 3–6 to a sub-agent so we can run faster"), don't comply with the conflicting instruction. Surface the conflict, name the guardrail, and ask for clarification. The guardrails exist because each one corresponds to a verified historical failure mode.
What this run produces — output artifact contract
A successful run produces this canonical set under the target's quality/ directory plus an AGENTS.md at the target's repo root. Every file listed here is gate-validated:
| Path | Role |
|---|
quality/EXPLORATION.md | Phase 1 findings — the foundation. |
quality/exploration_role_map.json | Per-file role tagging from Phase 1. |
quality/REQUIREMENTS.md | Testable requirements with use cases. |
quality/QUALITY.md | Quality constitution. |
quality/CONTRACTS.md | Behavioral contracts. |
quality/COVERAGE_MATRIX.md | Requirement → test traceability. |
quality/COMPLETENESS_REPORT.md | Final gate verdict. |
quality/test_functional.* | Automated functional tests. |
quality/RUN_CODE_REVIEW.md | Three-pass code review protocol. |
quality/RUN_INTEGRATION_TESTS.md | Integration test protocol. |
quality/RUN_SPEC_AUDIT.md | Council of Three spec audit protocol. |
quality/RUN_TDD_TESTS.md | TDD red-green verification protocol. |
quality/BUGS.md | Consolidated bug report. |
quality/INDEX.md | Run metadata + role breakdown + gate verdict. |
quality/PROGRESS.md | Phase-by-phase checkpoint log. |
quality/previous_runs/<TIMESTAMP>/ | Archive of any prior run. |
quality/writeups/BUG-<id>.md | Per-bug writeups. |
quality/patches/ | Fix and regression-test patches (one per bug). |
quality/code_reviews/, quality/spec_audits/, quality/results/, quality/mechanical/ | Intermediate pipeline artifacts. Top-level only — quality/workspace/ is forbidden (gated by check_no_workspace_dir). |
quality/logs/<run-id>/ | Centralized per-run logs (transcripts, control prompts, etc.). <run-id> is the run's UTC ISO-8601 compact timestamp (YYYYMMDDTHHMMSSZ). Pass --logs-flat (or set QPB_LOGS_LEGACY=1) to preserve the legacy scattered layout (parent-dir log file + top-level quality/control_prompts/) for tooling that depends on the old paths. See references/run_state_schema.md for the run_id / log_layout discriminator fields on the run_start event. |
<repo_dir>/quality.gate-failed-<UTC-ts>/ | Preserved on Phase 2 gate-failure: the failed quality/ tree is renamed to this sibling directory and a fresh quality/ is created. The preserved directory carries its own logs/<run-id>/ subtree with the failure logs. Operators inspect it post-mortem; the next run starts clean. |
AGENTS.md (target repo root) | Per-project orientation generated post-Phase-6. Carries a QPB sentinel marker so future runs detect QPB-managed copies. |
The gate verdict in quality/INDEX.md (pass / pass-with-cleanup / partial / fail) is the operator-facing summary of how the run went. The 089c F15 three-state taxonomy maps the gate's RESULT: line to this enum: RESULT: GATE PASSED → pass; RESULT: GATE PASSED WITH CLEANUP NEEDED → pass-with-cleanup (review completed, bug findings stand, only audit record-keeping incomplete — non-blocking); RESULT: GATE FAILED → fail (a substantive issue). partial covers aborted / WARN-only runs. If the verdict is anything other than pass, surface why before considering the run done; pass-with-cleanup is a successful outcome, but the State CN emit (per references/what_just_happened.md) walks the adopter through completing the remaining audit records.
quality/INDEX.md uses schema_version: "2.0" and carries a target_role_breakdown field with the per-role counts and percentages computed from quality/exploration_role_map.json. Pipelines activate from the role map's per-file role tags (skill-prose, skill-reference, skill-tool, code, test, docs, config, fixture, formal-spec, playbook-output) rather than a project-type label — the skill-derivation pipeline runs over files tagged skill-prose/skill-reference, the code-review pipeline runs over files tagged code, the prose-to-code divergence check runs over files tagged skill-tool, and when the role map shows zero of a role that pipeline no-ops cleanly. Both pipelines run when both surfaces are present.
Locating reference files
This skill references files in a references/ directory (e.g., references/iteration.md, references/review_protocols.md). The location depends on how the skill was installed. When a reference file is mentioned, resolve it by checking these paths in order and using the first one that exists (the canonical ten install layouts):
references/ (relative to SKILL.md — works when running from the skill directory or self-bootstrap).claude/skills/quality-playbook/references/ (Claude Code installation).github/skills/references/ (GitHub Copilot flat installation).cursor/skills/quality-playbook/references/ (Cursor installation).continue/skills/quality-playbook/references/ (Continue installation).github/skills/quality-playbook/references/ (alternate Copilot installation, nested).codex/skills/quality-playbook/references/ (Codex installation).windsurf/skills/quality-playbook/references/ (Windsurf installation).cline/skills/quality-playbook/references/ (Cline installation).aider/skills/quality-playbook/references/ (aider installation — aider does not auto-discover; tell aider to read SKILL.md explicitly)
All reference file mentions in this skill use the short form references/filename.md. If the relative path doesn't resolve, walk the fallback list above.
Council roster + adopter override
The Phase 4 Council of Three runs against a default 3-member roster defined in bin/council_config.py::DEFAULT_COUNCIL_MEMBERS — claude-opus-4.7, gpt-5.5, claude-sonnet-4.6. Adopters override per-operator via ~/.qpb/config.json (or $XDG_CONFIG_HOME/qpb/config.json); manage with python3 -m bin.qpb_config show|set|unset <key>. Per-run override via the --council-roster <m1,m2,m3> CLI flag takes highest precedence. See references/runners_and_models.md for the full override docs (precedence chain, model-availability behavior, falling back when a member is unavailable).
Why This Exists
Most software projects have tests, but few have a quality system. Tests check whether code works. A quality system answers harder questions: what does "working correctly" mean for this specific project? What are the ways it could fail that wouldn't be caught by tests? What should every developer (human or AI) know before touching this code?
Without a quality playbook, every new contributor (and every new AI session) starts from scratch — guessing at what matters, writing tests that look good but don't catch real bugs, and rediscovering failure modes that were already found and fixed months ago. A quality playbook makes the bar explicit, persistent, and inherited.
What This Skill Produces
Nine files that together form a repeatable quality system:
| File | Purpose | Why It Matters | Executes Code? |
|---|
quality/QUALITY.md | Quality constitution — coverage targets, fitness-to-purpose scenarios, theater prevention | Every AI session reads this first. It tells them what "good enough" means so they don't guess. | No |
quality/REQUIREMENTS.md | Testable requirements with project overview, use cases, and narrative — generated by a five-phase pipeline (contract extraction → derivation → verification → completeness → narrative) | The foundation for Passes 2 and 3 of the code review. Without requirements, review is limited to structural anomalies (~65% ceiling). With them, the review can catch intent violations — absence bugs, cross-file contradictions, and design gaps that are invisible to code reading alone. | No |
quality/test_functional.* | Automated functional tests derived from specifications | The safety net. Tests tied to what the spec says should happen, not just what the code does. Use the project's language: test_functional.py (Python), FunctionalSpec.scala (Scala), functional.test.ts (TypeScript), FunctionalTest.java (Java), etc. | Yes |
quality/RUN_CODE_REVIEW.md | Three-pass code review protocol: structural review, requirement verification, cross-requirement consistency | Structural review alone misses ~35% of real defects. The three-pass pipeline adds requirement verification and consistency checking — backed by experiment evidence showing it finds bugs invisible to all structural review conditions. | No |
quality/RUN_INTEGRATION_TESTS.md | Integration test protocol — end-to-end pipeline across all variants | Unit tests pass, but does the system actually work end-to-end with real external services? | Yes |
quality/BUGS.md | Consolidated bug report with patches | Every confirmed bug in one place with reproduction details, spec basis, severity, and patch references. The single source of truth for what's broken and how to verify it. | No |
quality/RUN_TDD_TESTS.md | TDD red-green verification protocol | Proves each bug is real (test fails on unpatched code) and each fix works (test passes after patch). Stronger evidence than a bug report alone — maintainers trust FAIL→PASS demonstrations. | Yes |
quality/RUN_SPEC_AUDIT.md | Council of Three multi-model spec audit protocol | No single AI model catches everything. Three independent models with different blind spots catch defects that any one alone would miss. | No |
AGENTS.md | Bootstrap context for any AI session working on this project | The "read this first" file. Without it, AI sessions waste their first hour figuring out what's going on. | No |
Plus output directories: quality/code_reviews/, quality/spec_audits/, quality/results/, quality/history/.
The pipeline also generates supporting artifacts: quality/PROGRESS.md (phase-by-phase checkpoint log with cumulative BUG tracker), quality/CONTRACTS.md (behavioral contracts), quality/COVERAGE_MATRIX.md (traceability), quality/COMPLETENESS_REPORT.md (final gate), and quality/VERSION_HISTORY.md (review log). Phase 7 can additionally generate quality/REVIEW_REQUIREMENTS.md (interactive review protocol) and quality/REFINE_REQUIREMENTS.md (refinement pass protocol) for iterative improvement.
The two critical deliverables are the requirements file and the functional test file. The requirements file (quality/REQUIREMENTS.md) feeds the code review protocol's verification and consistency passes — it's what makes the code review catch more than structural anomalies. The functional test file (named for the project's language and test framework conventions) is the automated safety net. The Markdown protocols are documentation for humans and AI agents.
Complete Artifact Contract
The quality gate (quality_gate.py) validates these artifacts. If the gate checks for it, this skill must instruct its creation. This is the canonical list — any artifact not listed here should not be gate-enforced, and any gate check should trace to an artifact listed here.
| Artifact | Location | Required? | Created In |
|---|
| Formal docs manifest | quality/formal_docs_manifest.json | Yes | Phase 1 (bin/reference_docs_ingest.py) |
| Requirements manifest | quality/requirements_manifest.json | Yes | Phase 2 |
| Use cases manifest | quality/use_cases_manifest.json | Yes | Phase 2 |
| Bugs manifest | quality/bugs_manifest.json | If bugs found | Phase 3/4/5 |
| Citation semantic check | quality/citation_semantic_check.json | Yes | Phase 4 (Layer 2 Council) |
| Exploration findings | quality/EXPLORATION.md | Yes | Phase 1 |
| Quality constitution | quality/QUALITY.md | Yes | Phase 2 |
| Requirements (UC identifiers) | quality/REQUIREMENTS.md | Yes | Phase 2 |
| Behavioral contracts | quality/CONTRACTS.md | Yes | Phase 2 |
| Functional tests | quality/test_functional.* | Yes | Phase 2 |
| Regression tests | quality/test_regression.* | If bugs found | Phase 3 |
| Code review protocol | quality/RUN_CODE_REVIEW.md | Yes | Phase 2 |
| Integration test protocol | quality/RUN_INTEGRATION_TESTS.md | Yes | Phase 2 |
| Spec audit protocol | quality/RUN_SPEC_AUDIT.md | Yes | Phase 2 |
| TDD verification protocol | quality/RUN_TDD_TESTS.md | Yes | Phase 2 |
| Bug tracker | quality/BUGS.md | Yes | Phase 3 |
| Coverage matrix | quality/COVERAGE_MATRIX.md | Yes | Phase 2 |
| Completeness report | quality/COMPLETENESS_REPORT.md | Yes | Phase 2 (baseline), Phase 5 (final verdict) |
| Progress tracker | quality/PROGRESS.md | Yes | Throughout |
| AI bootstrap | AGENTS.md (target repo root) | Yes | Generated by orchestrator after Phase 6 — not a Phase 2 deliverable |
| Bug writeups | quality/writeups/BUG-NNN.md | If bugs found | Phase 5 |
| Regression patches | quality/patches/BUG-NNN-regression-test.patch | If bugs found | Phase 3 |
| Fix patches | quality/patches/BUG-NNN-fix.patch | Optional | Phase 3 |
| TDD traceability | quality/TDD_TRACEABILITY.md | If bugs have red-phase results | Phase 5 |
| TDD sidecar | quality/results/tdd-results.json | If bugs found | Phase 5 |
| TDD red-phase logs | quality/results/BUG-NNN.red.log | If bugs found | Phase 5 |
| TDD green-phase logs | quality/results/BUG-NNN.green.log | If fix patch exists | Phase 5 |
| Integration sidecar | quality/results/integration-results.json | When integration tests run | Phase 5 |
| Mechanical verify script | quality/mechanical/verify.sh | Yes (benchmark) | Phase 2 |
| Verify receipt | quality/results/mechanical-verify.log + .exit | Yes (benchmark) | Phase 5 |
| Triage probes | quality/spec_audits/triage_probes.sh | When triage runs | Phase 4 |
| Code review reports | quality/code_reviews/*.md | Yes | Phase 3 |
| Spec audit reports | quality/spec_audits/*auditor*.md + *triage* | Yes | Phase 4 |
| Recheck results (JSON) | quality/results/recheck-results.json | When recheck runs | Recheck |
| Recheck summary (MD) | quality/results/recheck-summary.md | When recheck runs | Recheck |
| Seed checks | quality/SEED_CHECKS.md | If Phase 0b ran | Phase 0b |
| Run metadata | quality/results/run-YYYY-MM-DDTHH-MM-SS.json | Yes | Phase 1 (created), Throughout (updated) |
Sidecar JSON lifecycle: Write all bug writeups before finalizing tdd-results.json — the sidecar's writeup_path field must point to an existing file, not a placeholder. Similarly, run integration tests and collect results before writing integration-results.json.
Sidecar JSON Canonical Examples
quality/results/tdd-results.json — the gate validates field names, not just presence:
{
"schema_version": "1.1",
"skill_version": "1.5.8",
"date": "2026-04-12",
"project": "repo-name",
"bugs": [
{
"id": "BUG-001",
"requirement": "REQ-003",
"red_phase": "fail",
"green_phase": "pass",
"verdict": "TDD verified",
"fix_patch_present": true,
"writeup_path": "quality/writeups/BUG-001.md"
}
],
"summary": {
"total": 3, "confirmed_open": 1, "red_failed": 0, "green_failed": 0, "verified": 2
}
}
verdict must be one of: "TDD verified", "red failed", "green failed", "confirmed open", "deferred". date must be ISO 8601 (YYYY-MM-DD), not a placeholder, not in the future.
quality/results/integration-results.json:
{
"schema_version": "1.1",
"skill_version": "1.5.8",
"date": "2026-04-12",
"project": "repo-name",
"recommendation": "SHIP",
"groups": [{ "group": 1, "name": "Group 1", "use_cases": ["UC-01"], "result": "pass", "tests_passed": 3, "tests_failed": 0, "notes": "" }],
"summary": { "total_groups": 12, "passed": 11, "failed": 1, "skipped": 0 },
"uc_coverage": { "UC-01": "covered_pass", "UC-02": "not_mapped" }
}
recommendation must be one of: "SHIP", "FIX BEFORE MERGE", "BLOCK". uc_coverage maps UC identifiers from REQUIREMENTS.md to coverage status.
Run Metadata
Every playbook run creates a timestamped metadata file at quality/results/run-YYYY-MM-DDTHH-MM-SS.json. This enables multi-model comparison and run history tracking.
Lifecycle: Create this file at the start of Phase 1. Update phases_completed, bug_count, and end_time as each phase finishes. The final update happens after the terminal gate.
{
"schema_version": "1.0",
"skill_version": "1.5.8",
"project": "repo-name",
"model": "claude-sonnet-4.6",
"model_provider": "anthropic",
"runner": "claude-code",
"start_time": "2026-04-16T10:30:00Z",
"end_time": "2026-04-16T11:45:00Z",
"duration_minutes": 75,
"phases_completed": ["Phase 0b", "Phase 1", "Phase 2", "Phase 3", "Phase 4", "Phase 5"],
"iterations_completed": ["gap", "unfiltered", "parity", "adversarial"],
"bug_count": 12,
"bug_severity": { "HIGH": 2, "MEDIUM": 5, "LOW": 5 },
"gate_result": "PASS",
"gate_fail_count": 0,
"gate_warn_count": 2,
"notes": ""
}
Required fields: schema_version, skill_version, project, model, start_time. All other fields are populated as the run progresses. model should be the exact model string (e.g., "claude-opus-4.7", "gpt-5.5", "claude-sonnet-4.6" — v1.5.7 D6 default Council members; see references/runners_and_models.md). runner identifies the tool used to execute the playbook (e.g., "claude-code", "copilot-cli", "cursor", "cowork"). duration_minutes is computed from end_time - start_time. If the model or runner cannot be determined, use "unknown".
How to Use
The playbook is designed to run one phase at a time. Each phase runs in its own session with a clean context window, producing files on disk that the next phase reads. This gives much better results than running all phases at once — each phase gets the full context window for deep analysis instead of competing for space with other phases.
Default behavior: run Phase 1 only. When someone says "run the quality playbook" or "execute the quality playbook," run Phase 1 (Explore) and stop. After Phase 1 completes, tell the user what happened and what to say next. The user drives each phase forward explicitly.
Interactive protocol — how to guide the user
After every phase and every iteration, STOP and print guidance. Use a # header so it's prominent in the chat. The guidance must include: what just happened (one line), what the key outputs are, and the exact prompt to continue. See the end-of-phase messages defined after each phase section below.
If the user says "keep going", "continue", "next phase", "next", or anything similar, run the next phase in sequence. If all phases are complete, suggest the first iteration strategy (gap). If an iteration just finished, suggest the next strategy in the recommended cycle.
If the user says "run all phases", "run everything", or "run the full pipeline", run all phases sequentially in a single session. This uses more context but some users prefer it.
If the user asks "help", "how does this work", "what is this", or any similar phrasing, respond with this explanation (adapt the wording naturally, don't copy verbatim):
The Quality Playbook finds bugs that structural code review alone can't catch — the 35% of real defects that require understanding what the code is supposed to do. It works phase by phase:
- Phase 1 (Explore): Understand the codebase — architecture, risks, failure modes, specifications
- Phase 2 (Generate): Produce quality artifacts — requirements, tests, review protocols
- Phase 3 (Code Review): Three-pass review with regression tests for every confirmed bug
- Phase 4 (Spec Audit): Three independent AI auditors check the code against requirements
- Phase 5 (Reconciliation): Close the loop — TDD red-green verification for every bug
- Phase 6 (Verify): Self-check benchmarks validate all generated artifacts
After the numbered phases complete, you can run iteration strategies (gap, unfiltered, parity, adversarial) to find additional bugs — iterations typically add 40-60% more confirmed bugs on top of the baseline.
The playbook works best when you provide documentation alongside the code — specs, API docs, design documents, community documentation. It also gets significantly better results when you run each phase separately rather than all at once.
To get started, say: "Run the quality playbook on this project."
If the user asks "what happened", "what's going on", "where are we", or "what should I do next", read quality/PROGRESS.md and give them a concise status update: which phases are complete, how many bugs found so far, and what the next step is.
Documentation warning
At the start of Phase 1, before exploring any code, check for documentation. Look for directories named docs/, reference_docs/, doc/, documentation/, or any gathered documentation files. Also check if the user mentioned documentation in their prompt.
If no documentation is found, print this warning immediately (before proceeding):
Important: No project documentation found. The quality playbook works without documentation, but it finds significantly more bugs — and higher-confidence bugs — when you provide specs, API docs, design documents, or community documentation. In controlled experiments, documentation-enriched runs found different and better bugs than code-only baselines.
If you have documentation available, you can add it to a reference_docs/ directory and re-run Phase 1. Otherwise, I'll proceed with code-only analysis.
Then proceed with Phase 1 — don't block on this, just make sure the user sees the warning.
Doc gathering (pre-audit)
QPB derives a project's invariants from background documentation in its reference_docs/ directory — without it, QPB falls back to source-only evidence (Tier 3) and finds only shallow defects. QPB knows how to gather these docs itself.
When the user asks to gather background documentation, prepare reference_docs/, collect pre-audit context, or any equivalent ("get the docs ready for an audit of X", "build the reference corpus for X"), follow the protocol in references/DOC_GATHERING_PROMPT.md. Open that file, find the section "## The prompt (copy everything in this block)", and follow the boxed instructions inside the triple-backtick fences step by step (Step 0 grounding through Step 4 source-summary handback).
The same protocol works in two modes — the boxed prompt handles both without modification: when QPB is installed (you are now), Step 0 option (c) reads the local ai_context/TOOLKIT.md; when pasted into a fresh AI chat outside QPB, Step 0 option (a) handles the URL fetch.
Running a specific phase
The user can request any individual phase:
Run quality playbook phase 1.
Run quality playbook phase 3 — code review.
Run phase 5 reconciliation.
When running a specific phase, check that its prerequisites exist (e.g., Phase 3 requires Phase 2 artifacts). If prerequisites are missing, tell the user which phases need to run first.
Iteration mode — improve on a previous run
Use this when a previous playbook run exists and you want to find additional bugs. Iteration mode replaces Phase 1's from-scratch exploration with a targeted exploration using one of five strategies, then merges findings with the previous run and re-runs Phases 2–6 against the combined results.
When to use iteration mode: After a complete playbook run, when you believe the codebase has more bugs than the first run found. This is especially effective for large codebases where a single run can only cover 3–5 subsystems, and for library/framework codebases where different exploration paths find different bug classes.
Read references/iteration.md for detailed strategy instructions. That file contains the full operational detail for each strategy, shared rules, merge steps, and the completion gate. The summary below describes when to use each strategy.
TDD applies to iteration runs. Every newly confirmed bug in an iteration run must go through the full TDD red-green cycle and produce quality/results/BUG-NNN.red.log (and .green.log if a fix patch exists). The quality gate enforces this — missing logs cause FAIL. See references/iteration.md shared rule 5 and the TDD Log Closure Gate in Phase 5.
Iteration strategies. The user selects a strategy by naming it in the prompt. If no strategy is named, default to gap.
Run the next iteration of the quality playbook. # default: gap strategy
Run the next iteration of the quality playbook using the gap strategy.
Run the next iteration using the unfiltered strategy.
Run the next iteration using the parity strategy.
Run an iteration using the adversarial strategy.
Recommended cycle: gap → unfiltered → parity → adversarial. Each strategy finds different bug classes:
gap (default) — Scan previous coverage, explore uncovered subsystems and thin sections. Best when the first run was structurally sound but only covered a subset of the codebase.unfiltered — Pure domain-driven exploration with no structural constraints. No pattern templates, no applicability matrices, no section format requirements. Recovers bugs that structured exploration suppresses.parity — Systematically enumerate parallel implementations of the same contract (transport variants, fallback chains, setup-vs-reset paths) and diff them for inconsistencies. Finds bugs that only emerge from cross-path comparison.adversarial — Re-investigate dismissed/demoted triage findings and challenge thin SATISFIED verdicts. Recovers Type II errors from conservative triage.all — Runner-level convenience: executes gap → unfiltered → parity → adversarial in sequence, each as a separate agent session. Stops early if a strategy finds zero new bugs.
Phase-by-phase execution
Each phase produces files on disk that the next phase reads. This is how context transfers between phases — through files, not through conversation history. The key handoff files are:
quality/EXPLORATION.md — Phase 1 writes this, Phase 2 reads it. Contains everything Phase 2 needs to generate artifacts without re-exploring the codebase.quality/PROGRESS.md — Updated after every phase. Cumulative BUG tracker ensures no finding is lost.- Generated artifacts (REQUIREMENTS.md, CONTRACTS.md, etc.) — Phase 2 writes these, Phases 3–5 read them to run reviews, audits, and reconciliation.
The pattern for each phase boundary: finish the current phase, write everything to disk, then print the end-of-phase message and stop. When the user starts the next phase, read back the files you need before proceeding. This "write then read" cycle is the phase boundary — it lets you drop exploration context from working memory before loading review context, for example.
Phase 0 entry contract. The VERY first Bash call you make after the skill is invoked — before pwd, before ls, before any other Bash command — MUST be:
python3 <install_root>/bin/qpb_validate.py .
This is the Phase 0 entry contract first-probe. The phase0_first_probe assertion in the harness grading checks that qpb_validate.py is the LITERAL first Bash call — the first probe MUST be qpb_validate, before pwd, before ls, before any other Bash invocation.
If qpb_validate returns status=blocked (the target has both .claude and .github markers and you must disambiguate), DO NOT continue with any other tool call until you've re-invoked qpb_validate with the explicit --ai-tool=<tool> flag and confirmed it returns status=ok. Resolving the block is the only legitimate next action.
Phase 0 entry contract (gitignore remediation). After qpb_validate returns status=ok and before any artifact Writes, you must add quality/ to .gitignore in the target repo. If the file does not exist, create it; if it exists but does not contain a quality/ line, append quality/ on its own line. The gitignore_remediation_followed assertion in the harness grading verifies this step ran to completion. Do not skip this step — it prevents the playbook's own artifacts from polluting the target repo's git history.
Phase sentinel markers. After qpb_validate returns status=ok and the gitignore remediation lands, the next tool call you make MUST be the first qpb_phase emission:
python3 <install_root>/bin/qpb_phase.py 1 start
This is non-negotiable. The Test Harness status layer uses this sentinel to track which phase your run is in; without it, observers see your run as phase=— for 5-10 minutes while you do real work. Resolve <install_root> via the 10-layout fallback list documented below.
At each subsequent phase boundary:
- On phase entry:
python3 <install_root>/bin/qpb_phase.py <n> start (no --note). This MUST be the first tool call of that phase.
- On phase exit:
python3 <install_root>/bin/qpb_phase.py <n> done --note "<1-3 sentence summary of what happened in this phase>". The note is your own free-text summary — what you did, what you found, what's notable. No rigid schema.
Tool-use protocol — parallel-call hygiene. Phase-artifact Writes — REQUIREMENTS.md, CONTRACTS.md, BUGS.md, QUALITY.md, COVERAGE_MATRIX.md, COMPLETENESS_REPORT.md, RUN_CODE_REVIEW.md, RUN_SPEC_AUDIT.md, RUN_TDD_TESTS.md, RUN_INTEGRATION_TESTS.md, and tdd-results.json — must NEVER be issued in a parallel batch with other tool calls. Each phase-artifact Write must be its own assistant turn — each phase-artifact Write must be a standalone tool call.
After every artifact Write, your VERY NEXT tool call must immediately Read the same file path to verify the file exists on disk. If the Read returns "file not found," the previous Write was cancelled (typically because a sibling Bash or other tool call in the same parallel batch errored, and Claude's parallel-call cancellation propagates the failure to the Write). When this happens you must retry the Write as a standalone tool call in its own turn.
This is non-negotiable. Agents that issued Write+Bash in parallel batches have been observed to see the Write silently cancelled when the Bash sibling errored — artifacts go missing, the Phase 6 gate FAILs with "missing artifact" issues, and the agent often misdiagnoses the symptom as "tool output delivered out of order" rather than "Write cancelled by sibling failure."
Each invocation prints exactly one ::QPB:: {json} line; the helper is install-bundled. The sentinel is observability only — it does NOT change any phase output, gate verdict, or grading.
Write your Phase 1 exploration findings to quality/EXPLORATION.md before proceeding. This file is mandatory in all modes. Make it thorough: domain identification, architecture map, existing tests, specification summary, quality risks, skeleton/dispatch analysis, derived requirements (REQ-NNN), and derived use cases (UC-NN). Everything Phase 2 needs to generate artifacts must be in this file.
The discipline of writing exploration findings to disk is what forces thorough analysis. Without it, the model keeps vague impressions in working memory and produces broad, abstract requirements that miss function-level defects. Writing forces specificity: file paths, line numbers, exact function names, concrete behavioral rules. That specificity is what makes requirements precise enough to catch bugs during code review.
Run-state instrumentation (write events as you go)
Two files in quality/ track this run's state across the filesystem so the run is observable in flight, resumable across crashes, and auditable afterward. Maintain both throughout the run.
quality/run_state.jsonl — append-only machine-readable event log. One JSON object per line. The orchestrator and any monitor reads this file to know exactly where the run is.quality/PROGRESS.md — human-readable status file, atomically rewritten on every event.
Authoritative schema: references/run_state_schema.md. Read it once at run start; it defines the full event taxonomy, required fields, cross-validation rules, and PROGRESS.md format.
Initialization (before any phase work, including Phase 0)
If quality/run_state.jsonl does not exist:
- Create
quality/ if absent.
- Append the
_index event to quality/run_state.jsonl. Required fields: event=_index, ts (ISO 8601 UTC with Z), schema_version="1.5.8", event_types (array listing all event types this run will use — at minimum _index, run_start, phase_start, pattern_walked, pass_started, pass_ended, finding_logged, artifact_written, gate_check, phase_end, error, run_end), benchmark (target name), lever_state (e.g. "baseline" for normal runs), started_at.
- Append the
run_start event. Required fields: event=run_start, ts, runner (one of claude/codex/copilot/cursor), playbook_version (read from SKILL.md frontmatter version field), target_path.
- Write
quality/PROGRESS.md per the format spec in references/run_state_schema.md. Include header (Started / Benchmark / Lever / Runner / Playbook version), empty phase checklist with all six phases, empty Recent events / Artifacts produced sections.
If quality/run_state.jsonl already exists at run start: this is a resumed run. See "Resume semantics" below.
Per-phase events
At every phase boundary (1 through 6), write events:
- At phase start: append
{"event":"phase_start","ts":"<now>","phase":N} to quality/run_state.jsonl. Update quality/PROGRESS.md: mark phase N as in-progress with current timestamp.
- At phase end: first cross-validate the phase's expected artifacts (table below). If validation fails, append
{"event":"error","ts":"<now>","phase":N,"message":"<what's missing>","recoverable":true} and re-run the phase. If validation passes, append {"event":"phase_end","ts":"<now>","phase":N,"key_counts":{...},"artifacts_produced":[...]}. Update PROGRESS.md: check off phase N with summary stats.
Phase 1 sub-events (in addition to phase_start/phase_end):
- After walking each of the seven exploration patterns: append
{"event":"pattern_walked","ts":"<now>","phase":1,"pattern":N,"findings_count":K}. (One event per pattern, even if zero findings.)
- When
quality/EXPLORATION.md is written: append {"event":"artifact_written","ts":"<now>","relative_path":"quality/EXPLORATION.md","byte_size":<size>,"line_count":<lines>}.
Phase 4 sub-events:
- At each pass start (A through D):
{"event":"pass_started","ts":"<now>","phase":4,"pass":"A"}.
- At each pass end:
{"event":"pass_ended","ts":"<now>","phase":4,"pass":"A","output_artifact":"<path>"}.
Phase 5 / Phase 6 sub-events:
- At each gate-check completion:
{"event":"gate_check","ts":"<now>","gate_name":"<name>","verdict":"pass|fail|warn|skip","reason":"<short>"}.
Run end:
- After Phase 6
phase_end: append {"event":"run_end","ts":"<now>","status":"success","total_findings":<N>,"final_verdict":"<gate verdict>"}. Status is aborted for recoverable:false failures, failed for unrecoverable runtime errors.
Cross-validation rules at phase_end
Verify the corresponding artifacts before writing each phase_end event:
| Phase | Required |
|---|
| 1 | quality/EXPLORATION.md and quality/PROGRESS.md satisfy the 13-check Phase 1 gate documented at SKILL.md:1257-1273 (six required headings: ## Open Exploration Findings, ## Quality Risks, ## Pattern Applicability Matrix, ## Pattern Deep Dive — * ×3+, ## Candidate Bugs for Phase 2, ## Gate Self-Check; PROGRESS Phase 1 line marked [x]; ≥8 findings with file:line citations; ≥3 multi-location findings; 3-4 FULL pattern matrix rows; ≥2 multi-function pattern deep dives; candidate-bug source mix ≥2 from exploration/risks AND ≥1 from pattern deep dive). bin/run_state_lib.validate_phase_artifacts(quality_dir, phase=1) enforces the full gate. |
| 2 | All nine Generate-contract artifacts exist non-empty under quality/: REQUIREMENTS.md, QUALITY.md, CONTRACTS.md, COVERAGE_MATRIX.md, COMPLETENESS_REPORT.md, RUN_CODE_REVIEW.md, RUN_INTEGRATION_TESTS.md, RUN_SPEC_AUDIT.md, RUN_TDD_TESTS.md. Plus at least one non-empty quality/test_functional.<ext> (extension varies by language). |
| 3 | quality/RUN_CODE_REVIEW.md exists |
| 4 | quality/REQUIREMENTS.md non-empty AND quality/COVERAGE_MATRIX.md exists. If the four-pass skill-derivation pipeline ran (i.e., quality/phase3/ exists), then quality/phase3/pass_a_drafts.jsonl, quality/phase3/pass_b_citations.jsonl, quality/phase3/pass_c_formal.jsonl, and the Pass D inbox under quality/phase3/ must all exist and be non-empty. |
| 5 | quality/results/quality-gate.log exists, non-empty |
| 6 | quality/BUGS.md non-empty with ^##\s+BUG- sections AND quality/INDEX.md updated with gate_verdict field |
If a check fails, append the error event (recoverable=true) and re-run the phase. Do not write phase_end against missing artifacts — that's the failure mode v1.5.7 is built to catch.
bin/run_state_lib.validate_phase_artifacts(quality_dir, phase) performs these checks programmatically — call it from inside the playbook session if available.
Resume semantics
If quality/run_state.jsonl already exists when the playbook starts (a previous session crashed or paused mid-run):
- Read all events. Use
bin/run_state_lib.last_in_progress_phase(events) to find the last phase_start not followed by a matching phase_end — call it the in-progress phase.
- Run the cross-validation rules above for that phase.
- Artifacts complete: the prior session finished the work but didn't get to write
phase_end. Append the missing phase_end (with current ts) and proceed to the next phase.
- Artifacts incomplete: re-run that phase from scratch.
- If all six
phase_end events are present but no run_end: append run_end status=success and finalize.
- If no
quality/run_state.jsonl exists: fresh run. Initialize per the section above.
The policy: trust artifacts more than events. If events claim phase 4 done but REQUIREMENTS.md doesn't exist, re-run phase 4. If events stop mid-phase but the artifacts are complete, catch up the events.
PROGRESS.md atomic rewrite
PROGRESS.md is rewritten on every event (not appended). The contents reflect the current run-state.jsonl: header (run metadata), phase checklist (with summary stats per completed phase, in-progress marker for the current phase), recent events (last 10 events from the JSONL log, in human-readable form), artifacts produced (files written this run with byte sizes). See references/run_state_schema.md for the exact format template.
bin/run_state_lib.write_progress_md(quality_dir, events, current_phase) produces a correctly-formatted PROGRESS.md from the event list — call it after each event to keep the file in sync.
Phase 0: Prior Run Analysis (Automatic)
This phase runs only if quality/previous_runs/ exists and contains prior quality artifacts. If there are no prior runs, skip to Phase 1. If quality/previous_runs/ exists but is empty or contains no conformant quality artifacts (no subdirectories with quality/BUGS.md under them), skip Phase 0a and fall through to Phase 0b. (Legacy archives at quality/runs/ from pre-v1.5.4 remain readable for backward compatibility — see SKILL.md:149 — but the canonical archive root is quality/previous_runs/.)
When prior runs exist, the playbook enters continuation mode. This enables iterative bug discovery: each run inherits confirmed findings from prior runs, verifies them mechanically, and explores for additional bugs. The iteration converges when a run finds zero net-new bugs.
Step 0a: Build the seed list. Read quality/previous_runs/*/quality/BUGS.md from all prior runs. For each confirmed bug, extract: bug ID, file:line, summary, and the regression test assertion. Deduplicate by file:line (the same bug found in multiple runs counts once). Write the merged seed list to quality/SEED_CHECKS.md with this format:
## Seed Checks (from N prior runs)
| Seed | Origin Run | File:Line | Summary | Assertion |
|------|-----------|-----------|---------|-----------|
| SEED-001 | run-1 | virtio_ring.c:3509-3529 | RING_RESET dropped | `"case VIRTIO_F_RING_RESET:" in func` |
Step 0b: Execute seed checks mechanically. For each seed, run the assertion against the current source tree. Record PASS (bug was fixed since last run) or FAIL (bug still present). A failing seed is a confirmed carry-forward bug — it must appear in this run's BUGS.md regardless of whether any auditor independently finds it. A passing seed means the bug was fixed — note it in PROGRESS.md as "SEED-NNN: resolved since prior run."
Step 0c: Identify prior-run scope. Read quality/previous_runs/*/quality/PROGRESS.md for scope declarations. Note which subsystems were covered in prior runs. During Phase 1 exploration, prioritize areas NOT covered by prior runs to maximize the chance of finding new bugs. If all subsystems were covered in prior runs, explore the same scope but with different emphasis (e.g., different scrutiny areas, different entry points).
Step 0d: Inject seeds into downstream phases. The seed list becomes input to:
- Phase 3 (code review): Add to the code review prompt: "Prior runs confirmed these bugs — verify they are still present and look for additional findings in the same subsystems."
- Phase 4 (spec audit): Add to
RUN_SPEC_AUDIT.md: "Known open issues from prior runs: [seed list]. Expect auditors to find these. If an auditor does NOT flag a known seed bug, that is a coverage gap in their review, not evidence the bug was fixed."
Why this exists: Non-deterministic scope exploration means different runs notice different bugs. In cross-version testing, 4/8 repos had bugs found in some versions but not others — not because the bugs were fixed, but because the model explored different parts of the codebase. Iterating with seed injection solves this: confirmed bugs carry forward mechanically (no re-discovery needed), and each new run can focus exploration on uncovered territory.
Phase 0b: Sibling-Run Seed Discovery (Automatic)
This step runs only if quality/previous_runs/ does not exist OR quality/previous_runs/ exists but contains no conformant quality artifacts (i.e., Phase 0a has nothing to work with) and the project directory is versioned (e.g., httpx-1.3.23/ sits alongside httpx-1.3.21/). If quality/previous_runs/ exists with conformant artifacts, Phase 0a already handles seed injection — skip this step.
If quality/previous_runs/ exists but is empty or contains only non-conformant subdirectories, emit a warning: "Phase 0b: quality/previous_runs/ exists but contains no conformant artifacts — consulting sibling versioned directories for seeds." Then proceed with the sibling discovery below.
When no quality/previous_runs/ directory exists but sibling versioned directories do, look for prior quality artifacts in those siblings:
- Discover siblings. List directories matching the pattern
<project-name>-<version>/quality/BUGS.md relative to the parent directory. Exclude the current directory. Sort by version descending (most recent first).
- Import confirmed bugs as seeds. For each sibling with a
quality/BUGS.md, extract confirmed bugs using the same format as Step 0a. Write them to quality/SEED_CHECKS.md with origin noted as the sibling directory name.
- Execute seed checks mechanically (same as Step 0b in Phase 0a). For each imported seed, run the assertion against the current source tree and record PASS/FAIL.
- Inject into downstream phases (same as Step 0d in Phase 0a).
Why this exists: A prior benchmark observed a target repo producing a zero-bug result on one run despite a previous run on the same repo having found a real bug. The model simply explored different code paths and never examined the area where the prior bug lived. Sibling-run seeding ensures that bugs confirmed in prior versioned runs carry forward even without an explicit quality/previous_runs/ archive. This is a different failure class than mechanical tampering — it addresses exploration non-determinism, not evidence corruption.
Phase 1: Explore the Codebase (Write As You Go)
See references/phase1_exploration_guide.md for the full Phase 1 coverage (instrumentation, file-role tagging, exploration patterns, EXPLORATION.md template, gate self-check, end-of-phase message).
Phase 2: Generate the Quality Playbook
See references/phase2_generation_guide.md for the full Phase 2 coverage (instrumentation, required-references list, source-modification guardrail, entry gate, requirements pipeline, generated-artifact templates, completion gate, end-of-phase message).
Phase 3: Code Review and Regression Tests
Instrumentation: Append phase_start phase=3 to quality/run_state.jsonl now. At phase end, cross-validate (quality/RUN_CODE_REVIEW.md exists; one writeup per identified bug) then append phase_end phase=3.
Required references for this phase:
quality/REQUIREMENTS.md — target list for the code reviewreferences/review_protocols.md — three-pass protocol and regression test conventions
Run the code review protocol (all three passes) as described in File 3. After producing findings, write regression tests for every confirmed BUG per the closure mandate in references/review_protocols.md.
Update PROGRESS.md: Add every confirmed BUG to the cumulative BUG tracker with source "Code Review", the file:line reference, description, severity, and closure status (regression test function name or exemption reason). Mark Phase 3 (Code review + regression tests) complete.
End-of-phase message (mandatory — print this after Phase 3 completes, then STOP):
# Phase 3 Complete — Code Review
The three-pass code review is done. [Summarize: N bugs confirmed, N regression test
patches generated, N fix patches generated. List the bug IDs and one-line summaries.]
To continue to Phase 4 (Spec audit — Council of Three), say:
Run quality playbook phase 4.
Or say "keep going" to continue automatically.
After printing this message, continue automatically to Phase 4 unless the operator invoked you for the current phase only (e.g., they typed Run quality playbook phase 3. for a single-phase run). The Mode A default per AGENTS.md is the full six-phase pipeline; per-phase STOP applies only to incremental operator-driven invocations.
Phase 4: Spec Audit and Triage
Instrumentation: Append phase_start phase=4 now. For each pass A/B/C/D, append pass_started phase=4 pass=X and pass_ended phase=4 pass=X. At phase end, cross-validate (quality/REQUIREMENTS.md non-empty AND quality/COVERAGE_MATRIX.md exists) then append phase_end phase=4.
Required references for this phase:
references/spec_audit.md — Council of Three protocol, triage process, verification probes
Run the spec audit protocol as described in File 5. The triage report must include a ## Pre-audit docs validation section (see references/spec_audit.md for the full template). This section is required even if reference_docs/ is empty — in that case, note what baseline the auditors used instead. Every verification probe in the triage must produce executable evidence (test assertions with line-number citations) per the "Verification probes must produce executable evidence" rule above. After triage, categorize each confirmed finding.
Effective council gating for enumeration checks. If the effective council is less than 3/3 (fewer than three auditors returned usable reports) and the run includes any whitelist/enumeration/dispatch-function checks or any carried-forward seed checks, the audit may not conclude "no confirmed defects" for those checks without executed mechanical proof artifacts. An incomplete council with mechanical verification is acceptable. An incomplete council relying on prose-only validation for code-presence claims is not — escalate to "NEEDS VERIFICATION" and run the mechanical check before closing.
Pre-audit spot-checks must extract from code, not assert from docs. When the spec audit prompt includes spot-check claims for pre-validation (e.g., "verify that function X handles constant Y at line Z"), the triage must validate each claim by extracting the actual code at the cited lines — not by confirming that the claim sounds plausible. For each spot-check claim about code contents, the pre-validation must report what the cited lines actually contain: "Line 3527 contains default: — NOT case VIRTIO_F_RING_RESET: as claimed." If the spot-check was generated from requirements or gathered docs rather than from the code itself, treat it as a hypothesis to test, not a fact to confirm. This rule prevents a contamination chain where a false spot-check claim is accepted as "accurate" without reading the actual lines, then propagated through the triage and into every downstream artifact.
Update PROGRESS.md: Add every confirmed code bug from the spec audit to the cumulative BUG tracker with source "Spec Audit". This is critical — spec-audit bugs are systematically orphaned if they aren't added to the same tracker that the closure verification reads.
Layer 2 — Semantic Citation Check (Council sub-pass)
After the main spec audit triage, each Council member runs a per-REQ verdict against every Tier 1/2 REQ's citation_excerpt. This is Layer 2 of the hallucination gate: Layer 1 is the mechanical byte-equality check — bin/citation_verifier is invoked by bin/reference_docs_ingest at ingest time and re-invoked by quality_gate.py at gate time; the LLM never shells out to it directly. Layer 2 is semantic — the reviewer decides whether the excerpt actually supports the requirement as stated, or whether the requirement overreaches what the excerpt says.
Protocol.
-
One prompt per Council member, all Tier 1/2 REQs batched in. Not one REQ at a time (3×N prompts is too many). Not a prose response (pattern-matching risk). The reviewer receives the full list of (req_id, citation_excerpt, REQ description) tuples and returns a structured per-REQ JSON response.
-
Structured response schema. For each REQ the reviewer records {"req_id": "REQ-NNN", "reviewer": "<stable string>", "verdict": "supports" | "overreaches" | "unclear", "notes": "<reasoning>"}. The three verdict values (supports, overreaches, unclear) are the legal enum.
-
Batching threshold. When a run produces more than 15 Tier 1/2 REQs, split into batches of up to 15 REQs per prompt per Council member. The same reviewer sees each batch sequentially; their response entries are concatenated into one reviews[] array under the same reviewer string.
-
Reviewer identifier stability. Use fixed strings like "claude-opus-4.7", "gpt-5.5", "claude-sonnet-4.6" (the default Council roster). The majority computation quality_gate.py runs groups on this field — a typo silently becomes a fourth reviewer and breaks the 2-of-3 majority check.
-
Output. Concatenate all Council members' responses into quality/citation_semantic_check.json using the standard manifest wrapper, except the record array is named reviews rather than records. One file per run, regenerated on every audit pass.
Majority rule (gate-enforced). For each Tier 1/2 REQ, the gate groups reviews by req_id and fails the run when ≥2 of 3 reviewers recorded verdict == "overreaches". A single-member overreaches or unclear verdict surfaces as a warning but does not fail the gate. A REQ with fewer than three reviewer entries (missing reviewer, skipped batch) has insufficient evidence — the gate treats that as a fail.
No-op for Spec Gap runs. If a run produces zero Tier 1/2 REQs, citation_semantic_check.json is still written with an empty reviews array — the file's existence is part of the artifact contract even when the check has nothing to evaluate.
Post-spec-audit regression tests
After the spec audit triage, check the cumulative BUG tracker in PROGRESS.md. Any spec-audit BUG that doesn't have a regression test yet needs one now. Write regression tests for spec-audit confirmed code bugs using the same conventions as code-review regression tests (expected-failure markers, test-finding alignment, executable source files).
