メニュー
Use when the user wants to revalidate a running app against a spec, regenerate missing or weak tests, audit UI controls/forms/validation/dropdowns/API behavior/accessibility, or prove implemented features actually work after autospec-run.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
| name | autospec-qa |
| description | Use when the user wants to revalidate a running app against a spec, regenerate missing or weak tests, audit UI controls/forms/validation/dropdowns/API behavior/accessibility, or prove implemented features actually work after autospec-run. |
Run a spec-to-running-app QA audit, then regenerate missing or weak tests until
the application behavior is covered by executable evidence. This skill is the
explicit revalidation companion to autospec-test: autospec-test gates PRs
with deterministic coverage and E2E checks; autospec-qa performs a broad
human-style audit from the spec and turns gaps into stronger tests or follow-up
issues.
The per-area prose for this skill is also available as 8 cluster files under
skills/autospec-qa/clusters/<name>.md, dispatched by
qa-cluster-dispatch.sh (issue #730). The 8 canonical clusters are:
spec-traceability — extract spec requirements + traceability matrix.functional-coverage — UI controls, forms, validation, dropdowns, buttons.backend-integration — API contracts + no-mock smoke + live backend triage.reliability-contract — proof matrix, control ledger, mutation, canary,
no-mock coverage, observability, data lifecycle.legacy-and-cleanup — legacy removal gate + spec/docs cleanup +
deprecated-surface scan.benchmark-and-outsourcing — benchmark-overfit gate + outsourced-
implementation gate.accessibility-and-responsive — a11y + viewport + browser variance.production-incidents — incident registry regression check (PR #661).Each cluster carries only its area's prose plus the spec under audit. The
orchestrator (qa-cluster-dispatch.sh) is harness-aware (via
lib/autospec-harness-detect.sh, PR #725), honors
qa-cluster-coverage.sh for cross-cluster dedup, and honors
qa-verify-finding.sh for verify-first filtering. Aggregated findings
land in .autospec/qa-verdict.json, which the existing heal loop
(PR #666/#713) consumes unchanged.
The full per-area prose in the sections below remains canonical for now;
cluster files reference it and will be backfilled as the dispatch model
matures. Operators can opt out individual clusters via
qa-cluster-dispatch.sh --skip-cluster <name>.
/autospec-qa assumes the app under audit is already running at the URL the
operator supplies. For repos that need a fresh deploy plus a scoped test-data
clone before each QA cycle, the optional declarative contract
.autospec/qa-deploy.yml stands the app up first and records what/when/where
into the verdict — so a failed deploy surfaces as a single
code_health:qa_deploy_* finding instead of a downstream 502 triage trail.
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/qa-deploy-runner.sh" \
--repo-dir . --verdict .autospec/qa-verdict.json
is invoked at the very start of the orchestration — before any cluster fan-out — and its exit code gates the audit. The Live backend blocker triage prompt above stays as-is: it covers the deploy-succeeds-but-app-unhealthy-mid- audit case; the deploy runner covers the never-deployed case.
Contract file .autospec/qa-deploy.yml mirrors the declarative style of
.autospec/test.yml, parsed with yq -o=json and validated against
schemas/autospec-qa-deploy.schema.json with ajv (the same yq → jq → ajv
toolchain skills/autospec-test/scripts/load-contract.sh uses). Its presence
makes deploy mandatory. Key fields:
stages[] (required) — ordered deploy stages run sequentially under each
stage's timeout (default 300s); first failure aborts the rest. Each stage
has a name, a command run via bash -c, optional env, and an optional
health_check (url, expect_status default 200, retry default 30,
retry_interval default 10s) polled after the command.stages[].safety.max_records — required for any data-clone stage (name or
command matching clone|copy|replicate|sync).teardown[] — post-verdict cleanup entries (name, command, on_failure
warn|fail).target_envs.forbidden (required, ≥1) — the absolute safety floor.target_envs.allowed / safety.notes — advisory, recorded.Safety floor (non-negotiable, enforced regardless of operator config), each violation aborts immediately, runs no further stages, and exits 3:
target_envs.forbidden token (literal,
case-insensitive substring) matched against each stage command,
health_check.url, and captured stdout → qa_deploy_forbidden_target.command matched
word-boundary case-insensitively against --prod, --production, prod,
production, production-only, live-prod → qa_deploy_prod_pattern.max_records required for data-clone stages — a clone|copy|replicate| sync stage without safety.max_records → qa_deploy_missing_records_cap.Exit-code gating:
| Exit | Meaning | Verdict category |
|---|---|---|
| 0 | No .autospec/qa-deploy.yml (no-op), or all stages + probes succeeded | — / deploy block |
| 1 | A stage failed/timed out, or a health_check exhausted retries | code_health:qa_deploy_failed |
| 2 | Usage / contract invalid (ajv fail, malformed YAML, missing required) | code_health:qa_deploy_invalid_contract |
| 3 | Safety-floor violation | qa_deploy_forbidden_target | qa_deploy_prod_pattern | qa_deploy_missing_records_cap |
A failed deploy (exit 1/2/3) never reaches the cluster fan-out: the operator
sees a single code_health:qa_deploy_* finding naming the failing stage and a
stdout tail, not a 502.
deploy: verdict block. On success the runner writes a deploy: block into
.autospec/qa-verdict.json (atomically via the sibling .tmp + mv pattern
from qa-finding-filter.sh; a minimal {"deploy": {...}} is created
when no verdict exists yet, which the later audit merges into):
{
"deploy": {
"stages_run": ["deploy-to-test-family", "clone-production-data"],
"stage_durations_ms": {"deploy-to-test-family": 145000},
"target_env": "https://test.tidyboard.org",
"head_sha_deployed": "abc1234",
"data_clone": {"source": "prod-readonly", "records_copied": 8500, "max_records": 10000},
"teardown_run": true,
"teardown_failures": []
}
}
head_sha_deployed = git rev-parse --short HEAD at deploy time.
data_clone.records_copied is parsed from a records_copied=<int> stdout line
when present, else omitted. The block is additive — every existing verdict
consumer (heal loop, /autospec-release, /autospec-sweep) is unaffected.
Teardown. teardown[] entries run after the verdict is final, unless
--skip-teardown is passed (which sets deploy.teardown_run: false and leaves
the env intact for repeated manual re-QA). on_failure: warn logs only;
on_failure: fail downgrades a PASS verdict to PARTIAL (never touches
FAIL) and appends the entry to deploy.teardown_failures.
Graceful absent-file behavior. When .autospec/qa-deploy.yml is absent the
runner exits 0 immediately, writes nothing, and QA proceeds exactly as today —
the verdict is byte-identical with no deploy: key. This compatibility contract
is the most important one: behavior is byte-for-byte today's behavior when the
file is not present. Secret env values are never logged or recorded (only
key names). The runner uses set -u (not set -e) around stage execution so
failures are caught explicitly and recorded in the verdict.
If the feature-request argument matches update after trimming and lowercasing,
re-install the full autospec suite from main, show the before/after diff if the
harness exposes it, then stop. Do not run the QA audit.
| Capability | Claude Code | OpenCode | Codex CLI | Fallback if missing |
|---|---|---|---|---|
| Subagent model tier | Tier A: opus + ultrathink | Tier A: top-tier task + max reasoning | Tier A: current top GPT + reasoning_effort=high | Run inline, but keep the same report contract |
| Browser/E2E execution | Playwright/browser tool or shell | Playwright/browser task | shell + browser when available | Use browser-verified / fallback-smoke-only / not-run; mark UI-only gaps with blocker |
| Shell execution | Bash tool | shell tool | shell/apply_patch | Required for regeneration |
Model tier: TIER_A for the spec audit and test-regeneration plan because false PASS results are more expensive than extra review tokens.
Detect the harness once at skill start:
Agent with subagent_type is available.
TIER_A = opus + ultrathink.TIER_B = sonnet.task tool is available.
TIER_A = top-tier task model + high reasoning.TIER_B = smaller-tier task model + medium reasoning.apply_patch is the primary edit tool.
TIER_A = current top GPT + reasoning_effort=high.TIER_B = current cost-optimized Codex model + reasoning_effort=medium.Prefer a Tier A reviewer/QA subagent for the first traceability audit when the
harness supports it. If TIER_A is unavailable, silently fall back to the next
available top-tier model. If delegation is unavailable, run the audit inline.
/autospec-run finishes and the operator wants proof that the app
actually satisfies the source spec.autospec-test
contract explicitly allows scoped production and backup/restore is verified.For every user workflow button or dropdown-backed action, QA must cover both the contract and the deployed user-visible result:
502 bridge/proxy failures, and require either a working fallback route or a
clear actionable user state.If a required no-mock deployed/dev smoke path is NOT TESTED, the final verdict
must be PARTIAL unless the target repo's explicit test contract says that
workflow intentionally has no deployed smoke requirement.
Do not leave a no-mock smoke gap as a prose-only "remaining risk." For every
NOT TESTED no-mock smoke row, create or draft one of:
.autospec/test.yml, docs/**, or the repo's
equivalent QA contract explaining why deployed smoke is intentionally
unavailable for that workflow.If a dev URL and representative data are supplied, run the no-mock smoke test
instead of drafting setup work. If credentials or data are missing, keep the
traceability row NOT TESTED, keep the overall result PARTIAL, and make the
setup artifact the required next action.
Example chips, demo shortcuts, taxonomy tokens, conversion presets, and similar controls must be classified before the report is finalized:
decorative/presentational: assert that the element has no action contract,
no request side effect, and is either documented or clearly inert.workflow control: verify request contract, mocked happy path, no-mock smoke
path, success-state result, failure-banner absence, and known-fragile-backend
fallback behavior.Do not report presentational chips as a vague risk. Either prove they are intentionally decorative or file/draft the issue that wires and tests them as real controls.
Use this prompt every time the app creates, changes, tests, or audits any user-input element, including text boxes, textareas, selects, dropdowns, comboboxes, checkboxes, radio buttons, switches, date/time inputs, file inputs, search boxes, filters, sliders, segmented controls, editable tables, chips, or other controls that accept or change user intent:
For this user input element, answer before accepting the implementation or test:
1. What is this control for?
2. Which user goal or spec requirement does it serve?
3. How should a real user discover and use it?
4. What valid, invalid, empty, boundary, and recovery inputs must work?
5. What request payload, query, route, command, state change, persistence effect,
or visible result should change when the user changes this input?
6. What should happen when the backend rejects, times out, or partially accepts
this input?
7. Is this control intentionally read-only, disabled, decorative, or
presentational? If yes, where is that contract documented?
8. Did I implement and test this correctly, with evidence from the running app
rather than only DOM presence or mocked request dispatch?
Required outcome:
- If the control is functional, tests must cover discoverability, enabled/disabled
state, valid input, invalid input, boundary input, request/output effect,
success state, error state, and recovery.
- If the control is decorative or read-only, the QA report must classify it that
way and cite the spec, UI copy, or test contract that makes this intentional.
- If the answers are unclear, mark the traceability row AMBIGUOUS or PARTIAL and
draft the smallest spec/test/implementation follow-up needed to settle it.
When a no-mock deployed/dev smoke path is blocked by missing config, unknown
credentials, 502, 504, timeout, bridge, queue, worker, proxy, or backend
health errors, do not stop at "credentials unavailable" or "live backend not
tested" until you have made a bounded, non-secret-leaking config search.
For this live backend blocker, answer before finalizing the QA report:
1. Where does the app define its deployed/dev base URL, API endpoint, worker
endpoint, queue/bridge URL, and environment-specific service names?
2. Are these values available through safe config surfaces such as Terraform
outputs, deployment manifests, CI variables, documented env vars, cloud
resource names, or local `.env.example` files?
3. If credentials are required, which secret names or credential sources are
referenced, and can the smoke test use them without printing secret values?
4. If temporary credential files or tokens are created for probing, were they
stored outside the repo, restricted, and removed after the probe?
5. Is the failure caused by missing credentials, stopped infrastructure,
unhealthy bridge/proxy, missing worker, wrong route mental model, or a live
handler/index path that times out?
6. Can a direct health/status/fleet/ping probe distinguish infrastructure
outage from a route-specific worker failure?
7. Which no-mock smoke paths now pass with real domain data, and which exact
route/subject still fails?
8. What operator action, infrastructure fix, worker fix, or fallback behavior is
required next?
Required outcome:
- Never print secret values in the QA report, logs, issue body, or test output.
- If credentials/config are discoverable, run the no-mock smoke instead of
leaving it `NOT TESTED`.
- If infrastructure was stopped or unhealthy and can be safely started by the
target repo's documented dev procedure, record the action and rerun the smoke.
- If broad health probes pass but one route/subject times out, classify the
blocker as a route-specific live worker/handler failure, not missing
credentials.
- Report passed live paths and the remaining failing path separately.
The QA pass should produce machine-readable proof artifacts whenever the target
repo permits writing under .autospec/. If the repo is read-only, draft the
artifact contents in the report.
Required artifacts:
.autospec/proof-matrix.json: one row per spec requirement with fields for
requirement_id, spec_reference, expected_behavior,
implementation_files, automated_tests, live_evidence, status,
blocker, and follow_up..autospec/reliability.yml: the generated app reliability contract,
including required live smoke workflows, representative inputs, seed/live data
sources, allowed mocks, required no-mock paths, health probes, expected
fallback behavior, forbidden production URLs, and forbidden false-green
conditions such as "PASS from DOM presence only.".autospec/control-intent-ledger.json: every discovered user input, button,
link, chip, dropdown, form, modal trigger, table action, and workflow control
classified as functional, read_only, decorative, or unclassified, with
intent, owning spec reference, selector/locator, expected effect, tests, and
evidence..autospec/mutation-proof.json: critical workflow assertions that were
intentionally broken or simulated and proved that the test suite fails when
the implementation regresses..autospec/canary-results.json: post-merge or deployed/dev no-mock canary
results for representative workflows when a deployed/dev environment exists.Validate written artifacts against the matching repository schemas when tooling is available:
schemas/autospec-proof-matrix.schema.jsonschemas/autospec-reliability.schema.jsonschemas/autospec-control-intent-ledger.schema.jsonschemas/autospec-mutation-proof.schema.jsonschemas/autospec-canary-results.schema.jsonUse validate-qa-artifacts.sh --repo <target-repo> when the autospec
repository helper is available. It validates present .autospec/* proof files
and fails malformed artifacts while warning about missing draft-only artifacts.
Hard gate: a QA result cannot be PASS if .autospec/proof-matrix.json has a
FAIL, PARTIAL, NOT_TESTED, AMBIGUOUS, or unblocked unclassified row.
Every proof artifact must carry freshness metadata:
Before accepting an artifact, compare its freshness metadata to the current
checkout and tested environment. If the spec, implementation, app URL, or test
command changed after the artifact was generated, mark the artifact stale and
the affected rows PARTIAL until regenerated.
Every QA run invokes /autospec-doc --full (via
skills/autospec-doc/scripts/doc-orchestrator.mjs --full), then runs
verify-examples.mjs across the regenerated pages, then runs
check-doc-drift.sh --working-tree expecting a clean working tree.
documentation.auto_regenerate is NOT consulted here. QA is the guaranteed
sync point for documentation correctness, independent of the operator-controlled
auto-regeneration switch.
Each of the following failure classes is a QA finding in the standard QA report format, subject to the same severity discipline as the no-mock smoke rule (never silently skipped):
doc-orchestrator.mjs --full exits non-zero or
produces no output pages.verify-examples.mjs reports one or more example
failures across the regenerated audience pages.check-doc-drift.sh --working-tree exits non-zero,
indicating that generated pages differ from what is committed.Failures are reported as findings with release_blocking: true and category: documentation_gate, mirroring the no-mock smoke handling.
Graceful skip (logged as INFO, not a finding): when the target repo has no
documentation: config (orchestrator exits 2), the gate records an INFO log
line — docs: no documentation config found (exit 2), gate skipped — and
continues without emitting a finding. No other skip condition is permitted.
After QA proof artifacts are written, run the shared read-only quality audit so the QA report carries repo-wide code-quality, verification, dependency, route, security, and maintainability gaps discovered during revalidation:
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/repo-quality-audit.sh" \
--repo . \
--json ".autospec/repo-quality-audit.json" \
--markdown ".autospec/repo-quality-audit.md" \
${AUTOSPEC_QUALITY_AUDIT_FILE_ISSUES:+--file-issues}
Include the audit status, filed issue links, suppressed findings, and unfiled
residual risks in the final QA verdict. The helper is read-only with respect to
source files; it only writes audit artifacts and optional deduplicated GitHub
issues when operator policy permits issue creation.
Every PASS must cite provenance that another reviewer can inspect:
No provenance means no PASS. Request dispatch alone, DOM presence alone, a
generic result panel change, or a mocked-only response is not sufficient
provenance for backend-backed behavior.
For each browser or deployed/dev workflow, inspect console and network signals. The workflow fails unless the error is the explicit negative path under test:
Record expected negative-path failures separately from unexpected errors.
Before implementation, test generation, or final QA, scan the spec and current UI/API behavior for contradictions:
Contradictions become AMBIGUOUS rows with a proposed resolution and a
spec/config follow-up before implementation is called complete.
When two specs in docs/specs/ overlap on a behavior, the spec whose
last-modifying commit on the current branch is most recent wins (issue #635:
implicit-by-recency supersession; untracked specs fall back to filesystem
mtime). Operators do NOT write Supersedes: frontmatter — recency alone
decides.
QA validation MUST resolve each behavior to its authoritative spec before
classifying a row as FAIL. Behavior present only in an older, superseded
spec is not a QA failure — it has been replaced by the newer spec. Use the
shared helper to look up the authoritative spec for each behavior key:
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/resolve-spec-supersession.sh" "<behavior-key>"
For each proof-matrix row whose expected behavior overlaps multiple specs:
spec_reference, the row is being validated against a superseded
behavior. Re-derive the expected behavior from the resolver's winning
spec and update the row's spec_reference accordingly.superseded with a note pointing at the winning spec; do
NOT classify it FAIL.This rule applies symmetrically to control-intent ledger entries and live-evidence checks: a control whose intent is only described by a superseded spec is not a regression.
For CRUD, stateful, or workflow apps, prove the data lifecycle:
If the app has persistence but lacks lifecycle evidence, keep the affected
feature PARTIAL.
Generated apps must expose enough safe diagnostics to debug failed QA and canaries:
Missing observability does not automatically fail a pure UI feature, but it is a blocking gap for backend-backed workflows whose failures cannot otherwise be triaged.
Do not accept "green after retry" as reliable without classification. If a test uses retries, sleeps, widened timeouts, or repeated reruns to pass, classify the flake as one of:
Then either fix it, quarantine it with a follow-up issue and owner, or keep the
affected proof row PARTIAL.
Keep looking for reliability gaps beyond the obvious UI happy path:
Before finalizing, ask: "What else could make this generated app unreliable, duplicated, or falsely green against the spec?"
Repeat until the next answer is only:
.autospec/reliability.yml.Each actionable answer must become a strengthened test, implementation fix, proof-artifact update, reliability-contract update, or follow-up issue.
Before declaring a generated app reliable, create or validate
.autospec/reliability.yml:
For this app, define the reliability contract:
1. Which user workflows are required live smokes?
2. Which representative inputs and seed/live data prove each workflow?
3. Which tests may mock APIs, and which workflows require no-mock deployed/dev
execution?
4. Which health probes distinguish app, API, queue, bridge, worker, cache,
database, and external-service failures?
5. Which backend failures require fallback behavior, and what visible state
proves the fallback worked?
6. Which production or unsafe URLs are forbidden during tests?
7. Which false-green patterns are forbidden, such as DOM-only PASS, request-only
PASS, mocked-only PASS, or screenshot-only PASS?
Required outcome:
- If the contract is missing, draft it.
- If the contract conflicts with the spec, mark the affected rows PARTIAL and
file a spec/config follow-up.
- If no deployed/dev environment exists, require an explicit contract entry
explaining the blocker and exact setup action.
The UI audit must not rely on a vague list of "controls covered." Build a control intent ledger:
For every reachable control, record:
- selector or stable locator
- visible label and accessible name
- control type
- classification: functional, read_only, decorative, or unclassified
- spec requirement served
- expected request/state/persistence/result effect
- validation and error behavior
- tests proving the effect
- live no-mock evidence when required
- follow-up issue or blocker if not fully proven
Required outcome:
- Functional controls need request/output/result/failure coverage.
- Read-only and decorative controls need an explicit documented reason.
- Any unclassified control makes the QA result PARTIAL or FAIL.
Before accepting generated code or regenerated tests, challenge duplication:
For each new component, function, module, API client, validator, form helper,
error handler, backend wrapper, fixture, or test utility:
1. Search the repo for existing helpers or patterns that already solve this.
2. If a duplicate exists, reuse it or explain why it is insufficient.
3. If the new code remains, identify its owning spec requirement and public
contract.
4. Block duplicate API clients, duplicate validators, duplicate form state
logic, duplicate error banners, duplicate request wrappers, and duplicated
test fixtures unless the issue/spec explicitly requires a fork.
Required outcome:
- Duplication findings become implementation fixes or `DUPLICATE_CODE`
follow-up issues with file references.
For every critical user workflow, prove that the test would fail if the product were broken:
For this workflow test, simulate or reason through one minimal break:
1. Break request payload/query propagation.
2. Break the visible result rendering.
3. Return a known backend failure such as `502`, `504`, timeout, empty data, or
validation rejection.
4. Disable the fallback route or error banner.
5. Remove persistence or state update.
Required outcome:
- At least one critical workflow per feature has mutation/breakage evidence.
- If a test still passes after the behavior is broken, strengthen the assertion
before accepting the QA result.
- Record the result in `.autospec/mutation-proof.json` or the QA report.
For every test that consumes a public dataset, benchmark, leaderboard, or externally-curated answer key (e.g., ClassyFire, ChEMBL, MassBank, MNIST, GLUE, SuperGLUE, ImageNet, BIG-bench, HELM, GSM8K, MMLU, fixtures derived from a named corpus), audit for benchmark leak and overfit-shaped assertions:
For each test fixture and assertion derived from a public benchmark or
externally-curated dataset:
1. Identifier hard-coding — does the test pin specific benchmark IDs
(InChIKeys, accession numbers, row hashes, sample indices, image filenames,
prompt IDs) and assert exact downstream values?
2. First-N slice — is the fixture the first N rows / top-N records / a fixed
prefix of a known benchmark, rather than a randomized or rotated sample?
3. Answer-key shape — do the assertions match the benchmark's published
answer key (labels, class names, expected outputs) such that the system
under test could pass by memorizing those rows?
4. Distributional sibling — is there a randomized or property-based test
covering the same code path on a fresh draw from the source distribution?
5. Refresh path — can the fixture be regenerated on demand from the upstream
source (API, query, seeded sampler), or is it frozen indefinitely?
6. Training-on-test — is the same fixture also used to train, tune, select,
or prompt-engineer the code under test? If yes, the fixture is leaked.
Required outcome:
- Replace fixed slices with a seeded random sample drawn from the upstream
source per run (or per release), and rotate the seed at a defined cadence.
- Replace exact-value assertions on benchmark identifiers with property-based
or invariant assertions that hold for any valid draw from the distribution.
- Keep at most one tiny "shape smoke" test with hard-coded IDs, clearly named
as a smoke test, never as the primary correctness signal.
- A test that asserts exact benchmark answers without a randomized sibling is
a PARTIAL finding. If the same fixture is also used as the training or
selection signal for the code under test, it is FAIL (training-on-test
leak).
- Record each finding in the QA report under "Benchmark Overfit and Test
Leak" with the test path, the leaked dataset, the failure mode (1-6
above), and the required remediation.
When the spec uses an implementation verb (classify, resolve, score, parse, transform, extract, generate, rank, detect, segment, embed, …) and the diff or repo adds a runtime call to a third-party API/SDK that performs that same verb, the LLM has outsourced the product to an external service instead of implementing it. Audit every new or expanded external-service binding:
For each new HTTP client, SDK call, or service binding introduced to satisfy
a spec acceptance criterion:
1. Verb alignment — does the spec's verb describe an action the product is
supposed to PERFORM (build, classify, resolve, score, parse, transform,
extract, generate, detect, segment, embed) or merely INTEGRATE WITH
(fetch from, sync, import, mirror, publish to)? Implementation verbs
satisfied by remote-call shortcuts are outsourced implementations.
2. Benchmark / comparator collision — does the third-party service's name
appear elsewhere in `docs/specs/**`, README, or QA reports as a
"benchmark", "comparator", "ground truth", or "baseline"? Using the
benchmark at runtime contaminates evaluation and frequently violates
ToS.
3. Missing local algorithm — does the spec or referenced paper describe an
algorithm (rules, features, model, scoring formula) that the repo does
NOT implement, while a remote call answers the same question?
4. Cached external payloads — is there a `cache/`, `fixtures/`, or
`generated/` directory holding downloaded service responses that the
runtime depends on, with no in-repo computation able to reproduce them?
(Does NOT apply when the spec's verb is itself an integration verb
from signal 1's integration list — e.g., a project whose stated job is
to mirror/import/sync that external dataset.)
5. License and provenance — is there a provenance record (URL, fetch
timestamp, ToS link, redistribution scope) for every cached external
payload? Missing provenance on cached third-party data is a release
blocker even when the runtime call is itself legitimate.
6. Reimplementation directive — does the spec or product mission state
that the project is supposed to REPLACE or REIMPLEMENT the named
service? If yes, runtime use of that service is a definitional bug.
Required outcome:
- For each finding, name the spec acceptance criterion, the offending
client/URL/SDK call, and the signals (1-6) that matched.
- Remediation: replace the remote call with an in-repo implementation that
matches the spec's described algorithm. The external service MUST be
either (a) removed entirely, or (b) gated behind an off-by-default
feature flag scoped to evaluation/benchmark comparison only, with cached
payloads carrying explicit redistribution permission. No other
configuration is acceptable for release PASS.
- A finding matching signal 1 (implementation verb satisfied by remote
call), signal 2 (benchmark/comparator collision), or signal 6
(reimplementation directive) is FAIL — these are product-identity
violations and cannot be release-deferred.
- Signals 3, 4, or 5 alone default to PARTIAL until the in-repo
implementation lands. Signal 3 paired with signal 1 escalates to FAIL.
- Record each finding in the QA report under "Outsourced Implementation"
and in `.autospec/qa-verdict.json` with category
`outsourced_implementation`, the matched signal numbers in the summary,
and the remediation status.
The orchestrator soft-discipline sections above (the reliability exhaustion
loop, the 20-question Critical self-questioning checkpoint, "no-mock minimum
coverage", etc.) drive depth of coverage by prose. When clusters skip a step
the report can still say PASS while a production bug ships. The operator MAY
invoke /autospec-qa with one or more explicit machine-checkable flags that
convert that prose into a hard gate. With no flag, behavior is unchanged.
--every-spec-row — fail the run with code_health:incomplete_traceability
if any spec acceptance criterion lacks a non-NOT_TESTED row in
.autospec/proof-matrix.json. Forces the orchestrator to extract every
AC, not just the ones it judged "important".--mutation-each — require a mutation/breakage entry in
.autospec/mutation-proof.json for every workflow listed in the spec,
not just "critical" ones. Missing entries emit
code_health:incomplete_mutation_proof.--no-mock-each — require a no-mock deployed/dev smoke path in
.autospec/canary-results.json for every backend-backed feature.
Exceptions must be declared explicitly under no_mock_exempt in
.autospec/reliability.yml. Missing entries emit
code_health:incomplete_no_mock_coverage.--exhaustive — turns the three above on.Enforcement runs after the existing verdict computation but BEFORE writing
.autospec/qa-verdict.json as final. The helper appends one
release_blocking: true finding per gap and demotes the verdict from
PASS to PARTIAL (FAIL is preserved). The list of flags actually requested
is recorded under requested_flags on the verdict so downstream skills
(notably /autospec-release) can see what was enforced. Both the
orchestrator prompt and the regeneration loop call this helper:
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/qa-exhaustiveness-check.sh" \
--exhaustive \
--verdict .autospec/qa-verdict.json \
--proof-matrix .autospec/proof-matrix.json \
--mutation-proof .autospec/mutation-proof.json \
--canary .autospec/canary-results.json \
--reliability .autospec/reliability.yml \
--spec docs/specs/<feature>.md
Exit 0 means every requested gate passed. Exit 1 means at least one gate failed and the verdict + findings have been updated atomically. Backwards- compatible: invoking with no flag is a no-op.
/autospec-qa is self-healing by default. After discovery, the orchestrator
enters a discover → file → fix → re-QA loop and runs until convergence or a
guardrail trips. The loop is driven by qa-heal-loop.sh, which calls
qa-finding-to-issue.sh once per release-blocking finding to file an
auto-implement,autospec:v2-flow GitHub issue, then dispatches /autospec-run
to drain the queue before re-running QA.
Builds on the verify-first filter (#647 / PR #650): the heal loop relies on
the filter to reject unverified findings before they become issues, otherwise
round 1 would file noise. Builds on the production-incident registry
(#659 / PR #661): findings tagged
code_health:production_incident_regression_missing follow the same heal
path. Companion to autonomous mode (#662): when
~/.autospec/autonomous.flag is set, heal mode is implicit.
Loop semantics:
/autospec-qa, write .autospec/qa-verdict.json as today.release_blocking: true and
status ∈ {FAIL, PARTIAL, NOT_TESTED}:
qa-finding-to-issue.sh into a structured issue
body (category + summary + evidence + reproduction + remediation
directive verbatim from the relevant code_health:* directive map).auto-implement,autospec:v2-flow issue. Existing dedup:
skip if an open issue with the same (category, file, summary_hash)
already exists./autospec-run (single-agent Phase 4) to drain the new + existing
queue. One PR per issue per existing pipeline semantics.batch-done.json: ALL_DONE), re-enter step 1.Termination — loop exits when ANY of:
release_blocking: true. (Non-blocking findings are filed once but do
NOT keep the loop alive.) This is the canonical success state.(category, file, summary), equals round N's. The auto-fix
is going nowhere; the loop exits with oscillation_detected and
surfaces the persistent findings for operator review.AUTOSPEC_HEAL_MAX_ROUNDS reached (default 5).AUTOSPEC_HEAL_TOKEN_CAP (default
1.5M) or wall time > AUTOSPEC_HEAL_TIME_CAP (default 4h).verified_dropped round
after round, those findings are likely flaky probes; the loop exits
and asks the operator to triage.At loop end, the orchestrator prints a Markdown table to the operator and
writes .autospec/heal-summary.md for the audit trail and for
/autospec-release to consume as a release-readiness signal:
## /autospec-qa heal loop summary
| Round | Findings | Issues filed | PRs merged | Time | Status |
|-------|---------:|-------------:|-----------:|------|-------------------------|
| 1 | 7 | 7 | 6 | 12m | 1 finding persisted |
| 2 | 1 | 0 | 0 | 4m | oscillation_detected |
Final verdict: PARTIAL (1 persistent finding)
Invocation:
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/qa-heal-loop.sh" \
--verdict .autospec/qa-verdict.json \
--summary .autospec/heal-summary.md
Safety guardrails (NOT bypassed by heal mode) — inherit all existing autonomy/release guardrails:
autospec-autonomy-gate.sh).~/.autospec/no-heal.flag exists → heal globally disabled (operator
escape hatch for runaway loops).~/.autospec/stop.flag (graceful / immediate) terminates the loop at
the next round boundary.The heal loop runs by default (issue #711 locks the default-on contract; issue #666 introduced it). Four opt-outs revert to discovery-only behavior; all four are equivalent and only post-discovery behavior differs:
--no-heal flag.--single-pass flag (alias).~/.autospec/qa-no-heal.flag operator preference.~/.autospec/no-heal.flag legacy operator preference (still honored).When any of the four is present, /autospec-qa writes
.autospec/qa-verdict.json and exits without filing issues or
dispatching /autospec-run. Triggers are identical for both modes.
Backwards compatibility: any caller that relied on /autospec-qa
exiting after discovery must pass --no-heal. The
.autospec/qa-verdict.json schema is unchanged. Existing
/autospec-release and /autospec-sweep integrations see the post-heal
verdict, not the pre-heal one — this is the intended improvement.
qa-heal-loop.sh adopts the shared loop driver
lib/autospec-loop.sh (issue #708 / PR #712) for unified
termination naming. The four loop-enabled skills — /autospec --loop,
/autospec-refine --continue, /autospec-continue --loop, and
/autospec-qa (heal default-on) — share these termination conditions:
convergence_clean — zero release-blocking findings (heal alias:
convergence).oscillation_detected — finding-set hash repeats round-over-round.round_cap_reached — AUTOSPEC_HEAL_MAX_ROUNDS reached.evidence_based_stop — stop_marker / STOP: field in the QA
verdict.operator_stop — ~/.autospec/stop.flag or
~/.autospec/qa-heal-stop.flag (heal alias: stopped).budget_cap_reached — token or wall-time cap exceeded.Loop end produces .autospec/qa-heal-summary.md (mirrored from the
legacy .autospec/heal-summary.md path) with the same Markdown table
shape as .autospec/loop-summary.md (refine/continue/autospec --loop)
so /autospec-release and downstream consumers see structurally
identical summaries across all four loop-enabled skills.
When QA passes but production breaks, the same bug class tends to leak again the
next cycle because nothing in the QA pipeline learns from the incident. The
registry .autospec/production-incidents.json records every production incident
and the regression test that proves it cannot reoccur silently. /autospec-qa
reads this registry on every run and asserts, for each active incident:
regression_test file exists.expect(true).toBe(true) or empty
stub).mutation is applied to the
implementation — proving the test would catch a regression rather than
passing for unrelated reasons.Missing any of these four checks emits a finding under category
code_health:production_incident_regression_missing with
release_blocking: true in .autospec/qa-verdict.json.
Registry schema (one JSON array, one entry per incident):
[
{
"id": "INC-2026-05-28-checkout-double-charge",
"occurred_at": "2026-05-28T03:14:00Z",
"failure_mode": "concurrent submit creates duplicate Stripe charge",
"feature": "checkout/payment",
"spec_anchor": "docs/specs/checkout.md#payment",
"regression_test": "tests/regression/incident-2026-05-28-double-charge.test.ts",
"mutation": "remove the `submitting` guard on the Pay button; test must FAIL",
"added_by": "<operator handle>",
"status": "active"
}
]
status ∈ active | superseded | wont_test. Only active is enforced.
superseded means the underlying behavior was redesigned out of existence and
must cite the replacement spec section. wont_test requires explicit
justification (e.g., "infra-level only, no test can reach it") and emits a
warning, never a release blocker.
Never silently delete an incident entry. Operators move incidents to
superseded or wont_test with a citation/justification so the historical
record of why a regression test was added stays auditable. The orchestrator
runs the helper after the existing verdict computation, similar to other
post-verdict gates:
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/qa-incident-check.sh" \
--registry .autospec/production-incidents.json \
--verdict .autospec/qa-verdict.json
Exit 0 means every active incident is covered. Exit 1 means at least one
active incident is missing coverage and a release-blocking finding has been
appended to the verdict. The helper validates the registry against
schemas/autospec-production-incidents.schema.json and writes findings
atomically.
When a deployed/dev URL exists, define the post-merge canary before release:
For this feature, list the smallest deployed/dev workflow set that must run
after merge or deploy:
- route or screen
- representative input
- expected live API/backend path
- expected visible domain result
- failure banners that must not appear
- health probes to run if the canary fails
- issue label or escalation path for a failed canary
Required outcome:
- A missing canary for a backend-backed feature is PARTIAL unless the reliability
contract explicitly exempts it.
- A failed canary creates a regression issue with logs, route classification,
and live evidence.
Mocked tests are not enough for backend-backed features:
For every spec feature that depends on backend behavior, identify the minimum
no-mock path that proves real integration:
- one representative happy path,
- one known fragile backend path or fallback path,
- one visible success result,
- one absence-of-generic-failure assertion.
Required outcome:
- If no no-mock path exists, update `.autospec/reliability.yml` with the blocker
and setup action.
- If the app has only mocked coverage, do not mark the backend-backed feature
PASS.
Every new component, module, function, endpoint, worker, hook, or test helper must answer why it exists:
For this new code unit:
1. Which spec requirement or issue acceptance criterion owns it?
2. Why is an existing helper, component, service, or test utility not enough?
3. What is the smallest public contract this unit exposes?
4. What automated test and, when applicable, live workflow proves it?
5. What would break if this unit were wrong?
Required outcome:
- If there is no spec-linked reason, remove the code or file a scope/spec issue.
- If there is an existing helper, prefer reuse over a new abstraction.
When the current spec, operator direction, or accepted architecture deprecates a route, cache, bucket, storage backend, worker, feature flag, config key, UI path, or data source, autospec must remove the legacy surface and its spec/docs references instead of making the deprecated path work again.
For each legacy or deprecated surface:
1. What replaced it, and where is that replacement specified?
2. Which code paths, feature flags, env vars, infrastructure resources, docs,
tests, fixtures, runbooks, and specs still reference the legacy surface?
3. Did any smoke/test/repair step try to make the app pass by repopulating,
restarting, or relying on the deprecated surface?
4. What data migration, cleanup, or compatibility window is required before
removal?
5. Which tests prove the replacement path works and the legacy path is no longer
used?
6. Which specs or docs were updated to remove or mark the legacy behavior as
intentionally unsupported?
Required outcome:
- Do not populate deprecated caches, buckets, queues, indexes, or fallback stores
just to make a live smoke pass.
- If a deployed environment still depends on a deprecated surface, mark the row
PARTIAL/FAIL and file an implementation/infrastructure cleanup issue.
- Remove stale spec/docs references in the same change when implementation
removes legacy behavior.
- If compatibility must remain temporarily, document the sunset condition,
owner, and follow-up issue; tests must still prove the new canonical path.
Sweep already-merged code for the LLM-tier RULE_IDs STRING_MATCH_DOMAIN_LOGIC
and REPEATED_STRUCTURE_AS_CODE (AGENTS.md ## Implementation-quality contract).
This catches rot that pre-dates the PR-time guardian — substring-on-name
classifiers and ≥5-branch ladders that ship as a wall of if/elif/switch
instead of a table + dispatcher.
Supported languages: Python, JavaScript/TypeScript, Go, Java, Scala, Rust.
# Run from repo root. Emits findings into .autospec/qa-verdict.json under
# the category `code_health:brute_force_string_heuristics` and files one
# auto-implement issue per offender.
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/qa-brute-force-sweep.sh"
The sweep:
*.py, *.js, *.ts, *.jsx, *.tsx, *.go, *.java,
*.scala, *.rs file outside node_modules, .git, and dist.STRING_MATCH_DOMAIN_LOGIC: counts substring-style checks
(.contains(...), .includes(...), in name, strings.Contains,
etc.) and confirms the file imports a proper-representation library for
its language — Python rdkit/ast/urllib.parse/datetime/
ipaddress/lxml/jsonschema; JS/TS URL/Date/@babel/parser/
acorn/ts-morph/zod/ajv/joi; Go net/url/time/go/ast/
net.ParseIP/encoding/json; Java java.net.URI/java.time/
JavaParser/com.github.javaparser/javax.validation; Scala
java.net.URI/java.time/scalameta/refined/circe; Rust url::Url/
chrono/time/syn/std::net::IpAddr/serde.REPEATED_STRUCTURE_AS_CODE: counts repeated branch-shaped lines
(if/elif/case/match arms) and flags any file with ≥5..autospec/qa-verdict.json
under category code_health:brute_force_string_heuristics, carrying
rule_id, language, file, function, line.auto-implement,autospec:v2-flow GitHub issue per offender
via gh issue create. Issue body cites the file, function/method, line,
and the verbatim RULE_ID directive from AGENTS.md so the implementer
retry loop has the corrective instruction. Where the offending file has
no proper-representation library imported, the issue notes
"no proper-rep library imported — investigate first" so the implementer
does not blindly add a dependency.Errors in the sweep MUST NOT block the QA verdict; emit findings and
continue. End-to-end regression: after the QA-filed rewrite issues run
through /autospec-run, the offending files no longer trip either
RULE_ID in the PR-time guardian.
When .autospec/qa-verdict.json carries a release_blocking: true
finding whose category ends in :bug (or otherwise matches a
fingerprintable shape), run qa-bug-class-sweep.sh to fan out
across the repo for sibling instances of the same pattern class. Goal:
consistency by construction — the implementer fixes every matching
instance in one PR, not N PRs.
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/qa-bug-class-sweep.sh" \
--verdict .autospec/qa-verdict.json --repo-dir .
The sweep:
"", numbers →
N). The fingerprint and its sha256 (pattern_fingerprint,
fingerprint_hash) are stored on the PARENT finding at
finding-time, not recomputed — this gives pattern drift safety so
siblings remain resolvable even after the parent line is rewritten.git grep -nE <fingerprint> over the repo, excluding
node_modules/, .git/, and vendor/, plus the parent's own
file:line.(presence + density_bonus) * specificity where specificity scales
with fingerprint length (longer anchors → higher confidence). Matches
with confidence >= AUTOSPEC_QA_BUG_CLASS_MIN_CONFIDENCE (default
0.7) are appended to findings[] with
category: code_health:bug_class_sibling, parent_fingerprint,
confidence, file, line. Below-threshold matches land in
.autospec/qa-bug-class-flagged.json (operator review surface, NOT
filed as issues).qa-verify-finding.sh --category missing_function) to every
candidate sibling. If the bug is already fixed at HEAD for an
instance, that sibling is dropped silently..tmp + mv) so partial runs
cannot corrupt the verdict.Per-class issue filing: qa-heal-loop.sh groups findings by
parent_fingerprint when present and files one auto-implement issue
per pattern class carrying every instance's file:line. Findings
without a fingerprint stay per-finding as before. This is the key
contract that turns bug-class detection into a single coordinated fix.
Before any cluster emits a finding to .autospec/qa-verdict.json, the cluster
MUST verify that the symptom still reproduces at current HEAD. This eliminates
the dominant false-positive class observed in production: clusters re-flagging
gaps that landed in a recent PR. Run the shared probe per candidate finding:
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/qa-verify-finding.sh" \
--category <missing_function|missing_import|failing_test|spec_mismatch|regression> \
[--symbol <name>] [--file <path>] [--test-cmd <cmd>] \
[--spec <path>] [--impl <path>] [--claim <text>] [--commit <sha>]
Exit 0 means the finding still reproduces (KEEP and emit). Exit 1 means the
symptom no longer reproduces (DROP and append to verified_dropped[] in the
QA report). When ingesting prior-cycle findings from a previous
.autospec/qa-verdict.json, run the same probe; drops go into
stale_dropped[] rather than verified_dropped[].
Cluster cross-talk dedup: the QA orchestrator maintains
.autospec/qa-cluster-coverage.json via the shared coverage helper. Each
cluster MUST attempt to claim the (file, spec_section, rule_id) tuple it is
about to scan; a second cluster racing on the same tuple receives exit 1
("already claimed") and logs the skip as cluster_skip_dedup:
bash "${AUTOSPEC_SCRIPTS_DIR:-$HOME/.autospec/scripts}/qa-cluster-coverage.sh" \
claim --cluster <id> --file <path> --section <section> --rule <rule_id>
The final QA report MUST surface four explicit count fields so the operator
can see what the discipline filtered: findings_emitted, verified_dropped,
stale_dropped, cluster_skip_dedup.
After cluster fan-out completes, the QA orchestrator MUST run
qa-finding-filter.sh against the merged
.autospec/qa-verdict.json before any release-skill (e.g.,
/autospec-release) consumes it. The filter partitions findings into
verified, stale (re-probed and either refreshed or dropped to
verified_dropped[]), and unverified. Under AUTOSPEC_QA_STRICT=1
(the CI default), any finding lacking verified_at causes the QA run
to FAIL with status code_health:verify_first_violation; under
AUTOSPEC_QA_STRICT=0 unverified findings are routed to
unverified_warnings[] for operator triage. Cluster agents MUST emit
findings via qa-verify-finding.sh --emit-record <path> so the
required verified_at, verified_by, and verify_category fields are
populated by the probe rather than hand-constructed.
The QA orchestrator's cluster fan-out is a knob, not a constant. When the prior QA run dropped ≥30% of candidate findings on verification, prefer 3 verify-first clusters over 7 blind clusters for the next run. A drop rate >50% means clusters are re-litigating the same closed gaps rather than exploring new surface; shrink the fan-out and let each cluster do deeper verification work. This is prose guidance to the orchestrator — no enforcement script — but the rule is non-negotiable for runs whose prior-cycle drop rate exceeded the threshold.
When QA runs against the autospec repo itself, use dogfood-detectors.sh as the authoritative aggregated sweep rather than invoking individual scripts/qa-*-sweep.sh directly. The driver enforces an allowlist of intentional findings and fails on regression.
Before finalizing the QA report or accepting regenerated tests, stop and answer these questions in the working notes:
If any answer identifies a material gap, add the narrowest test for it or file a follow-up issue with reproduction steps. Do not silently leave it as an unstated risk.
Collect or infer:
If the app cannot be run and no URL is available, continue with static/test
audit only, but mark all runtime UI checks NOT TESTED with the exact blocker.
Use this prompt as the audit driver:
You are a senior QA engineer, product tester, and spec-compliance auditor.
Your job is to test this application completely from every meaningful scope and
direction, and to verify that every feature specified in the spec actually works
in the running app.
Inputs:
- Spec: <paste or link the product/spec document>
- App URL or local run command: <URL or command>
- Repository path, if available: <path>
- Test credentials/data, if available: <credentials/data>
Mission:
Create and execute a complete verification pass that proves whether the
implemented app matches the spec. Do not only inspect code. Run the app and
interact with it like a real user. Treat the spec as the source of truth.
Required testing scope:
1. Spec Traceability
- Extract every feature, requirement, acceptance criterion, user flow,
validation rule, permission rule, and edge case from the spec.
- Build a traceability matrix with requirement, expected behavior, test
performed, PASS/FAIL/PARTIAL/NOT TESTED result, evidence, and notes.
2. Functional Testing
Test every user-facing feature end to end: buttons, links, text inputs,
textareas, checkboxes, radio buttons, selects, dropdowns, multi-selects, search
fields, filters, sorting, pagination, tabs, modals, forms, uploads/downloads,
navigation, save/edit/delete flows, auth/login/logout, and empty/loading/
success/error/disabled states.
For every workflow button or dropdown-backed action, do not stop at clickability
or request dispatch. Verify the selected values change the real request
payload/query, run a mocked happy path, run a no-mock deployed/dev smoke path
with representative input, assert no live-failure banner appears, and assert a
real domain result is visible.
For every user input element, run the User Input Intent Prompt: identify what
the control is for, why it exists, how it should be used, what request/state/
visible result it should affect, how invalid or rejected input behaves, whether
it is intentionally decorative/read-only, and whether running-app evidence proves
it was implemented and tested correctly.
When live backend smoke is blocked by config, credentials, bridge/queue errors,
timeouts, or stopped services, run the Live Backend Blocker Triage Prompt before
declaring `NOT TESTED`. Search safe config surfaces, do not print secret values,
clean up temporary credential material, separate passed live paths from failing
routes, and classify whether the blocker is credentials, infrastructure health,
route mental model, or a route-specific worker/handler failure.
Create or draft the proof artifacts: `.autospec/proof-matrix.json`,
`.autospec/reliability.yml`, `.autospec/control-intent-ledger.json`,
`.autospec/mutation-proof.json`, and `.autospec/canary-results.json` when a
deployed/dev canary exists. Use them to prevent false PASS results: unclassified
controls, missing backend no-mock coverage, missing mutation proof, duplicated
code, or missing code intent keep the result PARTIAL or FAIL until resolved.
Validate artifacts against their schemas when possible, reject stale artifacts,
require evidence provenance for every PASS, fail unexpected console/network
errors, detect spec contradictions, prove data lifecycle for stateful features,
require observability for backend-backed workflows, classify flakes, and run the
Reliability Exhaustion Loop until all actionable reliability ideas are handled
or explicitly filed. Run the Legacy Removal and Spec Cleanup Gate whenever a
workflow touches a deprecated route, cache, bucket, storage backend, worker,
feature flag, config key, UI path, or data source.
3. Form and Validation Testing
For every form/input test valid input, empty input, invalid input, boundary
values, very long text, special characters, whitespace-only values, duplicates,
required-field behavior, error-message clarity, recovery after correction,
keyboard submission, disabled submit behavior, and server-side validation.
4. UI and UX Behavior
Verify text readability, labels, discoverability, visible state changes,
loading/error/success states, desktop/tablet/mobile layouts, and absence of
overlapping or clipped content.
5. Accessibility
Check keyboard-only navigation, focus order, visible focus indicators, input
labels, ARIA where needed, screen-reader names for icon buttons, color contrast,
Escape behavior, and Enter/Space behavior.
6. Data and State Testing
Verify persistence after refresh, unsaved-change handling, optimistic update
reconciliation, delete/edit propagation, filter/search/sort state, browser
back/forward behavior, and deep links.
7. Negative and Edge Cases
Test API failure, slow responses, empty datasets, large datasets, unauthorized
access, expired sessions, invalid routes, concurrent edits, repeated rapid
clicks, and refresh during in-progress actions.
8. Backend/API Integration
When a backend exists, verify frontend requests match API contracts, API errors
are handled correctly, storage reflects create/update/delete behavior, and
permissions/validation are enforced server-side.
Include regression coverage for deployed backend failures such as `502` bridge,
proxy, queue, or cache misses. If a backend route is known fragile, the UI must
either recover through a documented fallback route or show a clear actionable
state. Mocked endpoint success is not sufficient evidence.
9. Regression and Cross-Feature Testing
Combine flows: create then search/filter/sort, edit then verify list/detail
views, delete then verify navigation/empty states, change settings then verify
dependent screens, and complete full user journeys.
10. Automated Test Recommendations
Identify missing unit, integration, component, E2E, accessibility, and API
contract tests. For each missing test, state exactly what it should assert.
11. Reliability and Maintainability Proof
Validate the reliability contract, proof matrix, control intent ledger,
duplicate-code guardian findings, mutation/breakage proof, post-merge canary,
no-mock minimum coverage, and new-code intent gate. Treat these as required
spec-compliance evidence, not optional cleanup.
12. Freshness, Provenance, and Runtime Error Testing
Validate artifact freshness against the current commit/spec/environment, require
reviewable evidence provenance for every PASS, fail unexpected console/network
errors, classify retries/flakes, and record stale or unverifiable artifacts as
PARTIAL.
13. Spec, Data, Observability, and Exhaustion Testing
Detect spec contradictions, prove create/edit/delete/refresh/rollback/navigation
lifecycles for stateful features, verify observability for backend-backed
workflows, and run the Reliability Exhaustion Loop until no new actionable
quality ideas remain.
14. Legacy Removal and Spec Cleanup
Identify deprecated routes, caches, buckets, fallback stores, workers, config
keys, UI paths, and data sources. Verify the canonical replacement works, the
legacy path is not repopulated or relied on, stale specs/docs/tests are removed
or marked unsupported, and any temporary compatibility has a sunset issue.
Execution rules:
- Do not assume a feature works because code appears to exist.
- Do not mark PASS without concrete evidence.
- Mark untestable items NOT TESTED and explain the blocker.
- Mark spec mismatches FAIL even if the implementation seems reasonable.
- Mark ambiguous spec items AMBIGUOUS and propose expected behavior.
- Capture screenshots, logs, console errors, network failures, or command output
where useful.
- Continue until every spec item has a status.
Final report:
# QA Verification Report
## Summary
- Overall result: PASS / FAIL / PARTIAL
- App/version/commit tested:
- Environment:
- Main risks:
## Spec Traceability Matrix
| Requirement | Expected Behavior | Test Performed | Result | Evidence | Notes |
|---|---|---|---|---|---|
## Proof Artifacts
List `.autospec/proof-matrix.json`, `.autospec/reliability.yml`,
`.autospec/control-intent-ledger.json`, `.autospec/mutation-proof.json`, and
`.autospec/canary-results.json`, or state why each artifact was drafted inline
instead of written.
## Artifact Freshness and Evidence Provenance
List commit/spec/environment/test-command freshness checks and the exact
evidence cited for each PASS row.
## Runtime Error and Flake Classification
List console errors, network failures, runtime warnings, retries, sleeps,
timeouts, flake classification, and fixes or quarantines.
## Spec Contradictions
List contradictory or untestable requirements, UI/API/spec mismatches, proposed
resolution, and follow-up issue or spec edit.
## Data Lifecycle Proof
List create, refresh, edit, delete, rollback, idempotency, navigation, and
concurrency evidence for stateful workflows.
## Observability Contract
List correlation IDs, health probes, error boundary behavior, backend timing,
logs, user-safe messages, and any missing diagnostic hooks.
## Reliability Exhaustion Loop
List each "what else could make this unreliable, duplicated, or falsely green?"
answer and how it was handled.
## Legacy Removal and Spec Cleanup
List deprecated surfaces found, replacement path, code/config/docs/spec/tests
removed or updated, data cleanup or migration action, compatibility window if
any, and evidence that QA did not revive the legacy path.
## Bugs and Gaps
For each issue include severity, area, spec reference, steps to reproduce,
expected, actual, evidence, and suggested fix.
## UI Control Coverage
List every tested text input, select/dropdown, button, form, navigation element,
modal, table/list, and other control.
## Validation Coverage
List every validation rule tested, including valid, invalid, boundary, and
recovery behavior.
## User Input Intent Review
List every input-like control and answer: what it is for, why it exists, how a
user should use it, expected request/state/result effect, invalid/rejected-input
behavior, decorative/read-only status if applicable, and evidence that it was
implemented and tested correctly.
## No-Mock Deployed Smoke Coverage
List every deployed/dev smoke workflow, live URL/config used, representative
input/data, result, and blocker. Any required `NOT TESTED` no-mock smoke row
forces Overall result: PARTIAL unless the QA contract explicitly exempts it.
## Live Backend Blocker Triage
For each blocked live path, list safe config sources checked, secret names or
credential sources referenced without values, temporary credential cleanup,
health/status/fleet probes, passed real-data endpoints, remaining failing route
or subject, root-cause classification, and next operator or implementation
action.
## Presentational Control Classification
List each example chip, shortcut, preset, or token-like control as
decorative/presentational or workflow control, with the evidence or follow-up
issue/task.
## Duplicate-Code Guardian
List duplicated or near-duplicated components, helpers, validators, API clients,
request wrappers, error banners, fixtures, and test utilities, with reused
alternatives or follow-up fixes.
## Mutation and Breakage Proof
List critical workflow tests, the simulated break or mutation, expected failing
assertion, observed result, and strengthening performed if the test stayed green.
## Benchmark Overfit and Test Leak
List tests whose fixtures hard-code public benchmark identifiers, use a fixed
first-N slice, assert exact benchmark answer-key values, lack a randomized or
property-based sibling, or reuse the same fixture as training/selection signal.
For each: test path, leaked dataset, failure mode (1-6 from the gate), and
required remediation.
## Outsourced Implementation
List third-party API/SDK calls or cached external payloads where the spec
required the product to implement the action locally. For each: spec
acceptance criterion, offending client/URL, matched signal(s) from the gate
(1-6), and remediation (replace remote call with in-repo algorithm; keep
external service only as optional comparator behind a flag).
## Post-Merge Deployed Canary
List required canary workflows, representative input, deployed/dev URL, expected
domain result, result, and regression issue/action for failures.
## New Code Intent Gate
List new code units reviewed, owning spec/AC, reason existing helpers were not
enough, public contract, proof test, and what would break if wrong.
## Accessibility Findings
Include keyboard, focus, labels, contrast, ARIA, and screen-reader concerns.
## Cross-Browser / Responsive Findings
Include desktop, tablet, and mobile behavior if applicable.
## Automated Test Gaps
List exact tests to add before the app is fully protected.
## Final Verdict
State whether the app satisfies the spec and the minimum fixes required before
release.
After the audit:
FAIL, PARTIAL, NOT TESTED, and high-risk AMBIGUOUS
traceability row into one of:
NOT TESTED must produce
an environment setup issue/task, a QA contract update, or an operator-blocker
line with exact rerun instructions.
Any input-like control with an unanswered User Input Intent Prompt must
produce a strengthened test, implementation fix, spec clarification, or
follow-up issue.
A blocked live-backend row must run the Live Backend Blocker Triage Prompt
first and then produce either real no-mock smoke evidence or a classified
credentials/infrastructure/route/worker blocker.
Missing proof artifacts, unclassified controls, duplicated generated code,
missing mutation/breakage proof, missing canary, missing no-mock minimum
coverage, and unanswered new-code intent gates must each produce a
strengthened test, implementation fix, reliability contract update, or
follow-up issue.
Stale artifacts, missing evidence provenance, unexpected console/network
errors, unresolved spec contradictions, missing data lifecycle proof, missing
observability for backend-backed failures, unclassified flakes, and any
actionable Reliability Exhaustion Loop answer must also produce a fix,
stronger test, proof update, contract update, or follow-up issue.
Any deprecated surface still referenced by code, config, specs, docs, tests,
fixtures, infrastructure, or live-smoke repair steps must produce removal,
spec cleanup, migration work, or a sunset follow-up before PASS.At the end of every audit run (including aborted runs), write
.autospec/qa-verdict.json as the machine-readable result. Downstream
skills — notably /autospec-release — read this file as the authoritative
gate signal instead of re-parsing the prose report.
Schema:
{
"verdict": "PASS|PARTIAL|FAIL",
"head_sha": "<git rev-parse HEAD at run start>",
"generated_at": "<ISO-8601 UTC>",
"live_app_proof": true,
"artifacts": ["path/to/screenshot.png", "path/to/network-trace.har"],
"findings": [
{
"category": "benchmark_overfit|outsourced_implementation|no_mock_smoke|live_backend_blocker|mutation_proof_missing|spec_contradiction|duplicate_code|presentational_misclassified|legacy_residue|new_code_intent|accessibility|cross_browser|visual_fidelity|automated_test_gap|other",
"release_blocking": true,
"status": "PASS|PARTIAL|FAIL|NOT_TESTED",
"summary": "<one-line description>",
"evidence": "<path or pointer to artifact, report section, or issue>"
}
]
}
Rules:
live_app_proof is true ONLY if at least one interaction artifact
(screenshot, DOM snapshot, network trace, recorded session) was captured
against a booted app whose commit matches head_sha. Otherwise false.release_blocking defaults TRUE for:
benchmark_overfit with failure mode 6 (training-on-test leak),Use when the user wants to plan a feature — bootstrap repo if missing, brainstorm a design spec, decompose into linked GitHub issues, or split an existing spec into issues. Stops after Phase 3 and hands off to /autospec-run for autonomous implementation.
Use when the user has already run /autospec-define (or otherwise has a populated set of auto-implement GitHub issues) and wants the implementation half — Phases 4-6 — to run autonomously with admin auto-merge. Supports --profile <name> filtering against ~/.autospec/model-profiles.yml.
Use when the user wants /autospec-autonomous to run the autospec machinery unattended for weeks — a perpetual self-driving conductor that walks a never-idle priority waterfall (Tier 0 control channel + Tier 1 backlog→main + Tier 1.5 open-issue promotion + Tier 2 local discovery + Tier 3 architecture/coverage improvement + Tier 4 internet/operator discovery + capability-gated Tiers 5-7 growth outbound/define/measure), parks before quota exhaustion, obeys a GitHub control channel for live steering, and resumes automatically.
Observe a live autospec-autonomous conductor read-only by summarizing typed status, timeline, launch, heartbeat, tier, and discovery artifact surfaces.
Use when Autospec runtime provisioning detects Docker Compose files that require migration for isolated worktree environments.
Use when the user wants to ship a feature end-to-end across multiple commits — bootstraps repo if missing, brainstorms a design spec, decomposes into linked GitHub issues, or splits an existing spec into issues, then runs an autonomous implementation loop with auto-merge.