Use when the user wants end-to-end QA testing for a feature, branch, or PR — scans the project for entry points, generates a TEST_PLAN.md with edge cases, launches the service in an isolated container, and dispatches test runners (cheap agy for simple HTTP cases, claude for complex flows). Profile-driven: `init` once, then `run` repeatedly.
Installation
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Use when the user wants end-to-end QA testing for a feature, branch, or PR — scans the project for entry points, generates a TEST_PLAN.md with edge cases, launches the service in an isolated container, and dispatches test runners (cheap agy for simple HTTP cases, claude for complex flows). Profile-driven: `init` once, then `run` repeatedly.
x-qa — Profile-Driven E2E QA Orchestrator
Input Contract
run accepts a free-form {{ARGUMENTS}} — empty string, PR #<n>, an
entry-point name, a file path, a directory, or prose describing a feature.
The bootstrap classifier (Run Phase 3) resolves it. Explicit flags
(--pr/--branch/--service/--plan) override classification. Do NOT
add new user-facing flags for input source — the classifier handles it.
Bootstrap (MANDATORY)
Before any subcommand:
Pin capabilities per ../x-shared/capability-loading.md.
Read ../x-omo/SKILL.md if agy_cli or plugin.omc capability is pinned (needed for fanout).
Pin runner pair (D10) — derive X_QA_SIMPLE_RUNNER and X_QA_COMPLEX_RUNNER from the Capability Routing table below and export them for downstream phases. The runner template in references/case-runner-prompts.md reads these env vars rather than hardcoding a tool name.
Also pin X_QA_EXPLORE_MODE (team when team orchestration / plugin.omc is pinned, else bg-fanout) and X_QA_EXPLORER (the exploratory worker subagent) per the Exploratory Team Routing table.
For run only (D9) — invoke scripts/doctor.sh first. Refuse if the doctor fails. Skippable via --skip-doctor.
run flags: --worktree <path>, --service <name>, --channel <name>, --plan <path>, --no-launch, --max-bg <N> (default 8), --no-bg (force synchronous dispatch), --staleness-days <N> (default 7), --retry-flaky <N> (default 2), --allow-flaky-rate <pct>, --verdict-only, --skip-doctor, --no-profile, --pr <num>, --branch <name>, --no-kb (skip corpus + baseline + auto-promote), --kb-promote-after <N>, --kb-disable-auto-promote, --allow-coverage-gaps (downgrade the coverage gate to a warning), --no-explore (skip the exploratory bug-hunt) / --explore (force it even in CI)
KB Subcommands
The KB (.x-skills/x-qa/kb/) is the team-shared, git-tracked layer that
makes future runs faster and lets teams reuse proven test scripts.
Full schema: references/kb-schema.md. Curation rules:
references/kb-curation.md.
Form
Purpose
/x-skills:x-qa kb list [--cases|--flows|--baselines]
Run the judge over kb/evals/gold/<id>.jsonl, compute Cohen's κ, write kb/evals/calibration/<id>.json. Required before a judge can hard-fail a run. (Routes to calibrate-judge.sh --gold <repo>/.x-skills/x-qa/kb/evals/gold/<id>.jsonl --rubric-id <id> --out-dir <repo>/.x-skills/x-qa/kb/evals/calibration/ [--threshold 0.8].)
Cross-team sharing is the git-tracked KB itself (or a submodule when shared
across repos). Tarball export/import was cut as redundant with git.
Capability Routing
Pinned
Simple Runner
Complex Runner
agy_cli + plugin.omc
x-gemini (agy) serial
Agent oh-my-claudecode:qa-tester (sonnet) bg
agy_cli only
x-gemini (agy) serial
Agent Explore (sonnet) bg
plugin.omc only
Agent oh-my-claudecode:executor (haiku) bg
Agent oh-my-claudecode:qa-tester (sonnet) bg
neither
Agent Explore (haiku) bg
Agent Explore (sonnet) bg
The bootstrap step pins the runner pair into env vars X_QA_SIMPLE_RUNNER and X_QA_COMPLEX_RUNNER (read by case-runner-prompts.md). Never hardcode an Agent subagent_type in the runner template — always reference the pinned env var.
Serial-agy rule (MANDATORY). When X_QA_SIMPLE_RUNNER is the agy-agent (x-gemini) runner, simple-runner cases MUST be dispatched one at a time — effective --max-bg 1 for the agy runner — because concurrent agy processes hang (agy is a process-global singleton). The parallel-wave fanout (--max-bg) stays in effect for the non-agy fallback simple runners (OMC executor / Explore) and for all complex runners (qa-tester / Explore). The cap is on concurrent agy processes (≤1), not on the wave: complex cases and non-agy simple cases in the same wave still run in parallel; only the agy simple cases serialize among themselves.
Exploratory Team Routing (Arc C)
Pinned
X_QA_EXPLORE_MODE
X_QA_EXPLORER
team orchestration (plugin.omc)
team (shared bug-board)
oh-my-claudecode:qa-tester (sonnet)
otherwise
bg-fanout (background Agent)
Explore (sonnet)
Bootstrap pins X_QA_EXPLORE_MODE and X_QA_EXPLORER for Phase 13.5. See
references/explore-team.md (mode gate, bounded swarm) and
references/explorer-prompts.md (worker prompt).
Run Envelope (machine-readable, stable)
Success:
✓ x-qa run complete
QA_VERDICT=pass|warn|fail
QA_VERDICT_REASON=<one-line> # first blocking failure, or first warning, or "all gates passed"
QA_TOTAL=<n>
QA_PASSED=<n>
QA_FAILED=<n>
QA_FLAKY=<n>
QA_FLAKY_RATE=<float>
QA_REPORT=<abs path>
QA_PLAN=<abs path>
QA_RUN_ID=<id>
DURATION_S=<float>
ENTRY_POINT=<name>
SERVICE_LAUNCHED=true|false
KB_REUSED=<n> # cases pulled from corpus (not regenerated)
KB_GENERATED=<n> # cases minted this run
KB_PROMOTED=<n> # cases auto-promoted to corpus this run
KB_PROMOTE_STATUS=ok|disabled|error
COVERAGE_REQUIRED=<n> # required obligations from scope.json
COVERAGE_COVERED=<n> # required obligations satisfied by a case
COVERAGE_UNCOVERED=<csv> # uncovered required obligation ids ("" when none)
EXPLORE_RAN=true|false # false when skipped (CI / --no-explore / no service)
EXPLORE_FINDINGS=<n> # unique suspected findings on the bug-board
EXPLORE_CONFIRMED=<n> # findings that survived triage
EXPLORE_CASES_MINTED=<n> # confirmed findings minted into kb cases
EXPLORE_OBLIGATIONS_ADDED=<n> # novel obligations minted from "none" findings
CHANNELS_TESTED=<csv> # channels selected for execution (names)
CHANNELS_SKIPPED=<name:reason,...> # skipped channels + reason (stateful-not-owned / stateful-unverifiable / stateful-owned-chat-driver-deferred)
QA_VERDICT is ternary as of v2. warn = non-blocking gates failed, no blocking ones did. Consumers branching on pass|fail should treat warn as pass for back-compat OR opt into ternary semantics via QA_VERDICT_REASON.
Failure:
✗ x-qa run FAILED
REASON=<one-line>
PHASE=<bootstrap|doctor|plan|launch|dispatch|collect|aggregate>
QA_RUN_ID=<id-if-allocated>
doctor: references/doctor-checks.md + scripts/doctor.sh (validates both profile and kb/index.json)
generate: references/test-plan-schema.md (LLM-generated inline; scripts/plan-generate.sh is a v2 placeholder, not shipped in v1). Planner MUST consult kb/cases/ + kb/flows/ first per the corpus contract in references/test-plan-schema.md § Corpus Reuse.
run: see "Run Phases" below; intent via references/intent-detection.md; scout via references/scout-prompt.md; KB layer via references/kb-schema.md + references/kb-curation.md
kb <subcmd>: dispatched to scripts/kb-<subcmd>.sh. No LLM in the loop — these are pure-shell maintenance commands.
Run Phases
Bootstrap (above).
Memory recall (only when mcp.basic_memory pinned in bootstrap-active set): one mcp__basic-memory__search_notes({ query: "<test path or framework + 'flake'>", page_size: 10 }) call. Surface prior flake notes as test-history context for case classification — not autopilot. Apply consumer rules from ../x-shared/mcp-toolbox.md § Consumer rules. When mcp.basic_memory is not pinned, skip silently — Claude's native auto-memory file still applies.
Auto-doctor (skippable via --skip-doctor).
Classify intent. Run scripts/classify-intent.sh "{{ARGUMENTS}}", persist to <run-dir>/intent.json. If confidence == "low" OR multiple candidates surface, ask the user ONE question per references/intent-detection.md § Ask-When-Ambiguous, then rewrite intent.json with confidence: high.
Resolve target from intent: service → entry name; branch/pr → PR-surface derivation (references/pr-surface-derivation.md); spec/artifact/artifact-dir/prose → trigger Phase 5 (Scout). Refuse if resolved entry's type != http (v1 limitation).
Channel resolution. If intent.json.resolved.channel (or --channel)
is set, resolve it against profile.json.channels[] and pin
X_QA_CHANNEL + X_QA_DRIVER. Read the driver's feature-gate per
references/channel-drivers.md:
http → execute (Phases 8–15 as today, against the channel's base_url).
browser / computer-use → if the gating capability is absent, emit
CHANNEL_SKIPPED=<name> reason=driver '<driver>' not executable and stop
with a clear notice (capture-only in this release). Do NOT fall back to a
different channel silently.
Stateless-first default (no --channel). Run scripts/lib/channel-select.sh --profile <profile> --worktree <worktree> [--channel <name>] → persist
<run-dir>/channels.json. With no --channel, it defaults to stateless
channels (singleton_id == null) on the primary entry point. No channels[]
at all → the implicit primary http channel (back-compat).
Stateful resolution (ownership from feature-overrides.local.json only, R2):
owned here AND driver == http → EXECUTE via the existing http runner path.
owned here AND driver ∈ {browser, computer-use} → skip
CHANNEL_SKIPPED reason=stateful-owned-chat-driver-deferred.
not owned (the default) → skip CHANNEL_SKIPPED reason=stateful-not-owned.
isolate not set up → skip CHANNEL_SKIPPED reason=stateful-unverifiable
(never test a stateful channel blind).
channels.json.tested drives Phases 8-15; channels.json.skipped feeds the
envelope's CHANNELS_SKIPPED.
Scout (conditional). Only when intent ∈ {spec, artifact, artifact-dir, prose}: dispatch $X_QA_SIMPLE_RUNNER inline per references/scout-prompt.md. Persist <run-dir>/scope.json. On invalid JSON / timeout, use whole-profile coverage and warn. The scout also performs Domain Research (references/scout-prompt.md § Domain Research) — code-first modeling of entities/constraints/invariants/transitions — and emits domain_model + obligations[] into scope.json. When intent is not scout-eligible (branch/pr/service), obligations[] is absent and Phase 7.5 is a no-op.
KB consult (skipped on --no-kb). Read kb/index.json. For every endpoint in scope (from scout or PR-surface), collect matching kb/cases/*.yaml (not quarantined) and kb/flows/*.yaml. Pass this corpus to the planner.
6.5. Gap analyze (skipped on --no-kb). Run scripts/gap-analyze.sh --scope-file <run-dir>/scope.json when scout produced a scope (otherwise run without --scope-file and analyze the full index). Persist to <run-dir>/coverage_gaps.json. Inject as a ## Coverage Gaps block in the planner prompt per references/gap-analyzer.md. Default --staleness-days 7; override via profile.json.gap_analyzer.staleness_days.
Plan: read --plan <path> if given, else generate per references/test-plan-schema.md § Corpus Reuse using profile catalog + scope.json + KB corpus as ground truth. The planner SHALL prefer corpus IDs over fresh generation when the endpoint+category already has a green case. Cases requiring auth inherit profile.json.auth_case_id as a default precondition_case_id per references/kb-schema.md § precondition_case_id and references/case-runner-prompts.md § Precondition Chaining.
7.5. Coverage gate (skipped when scope.json has no obligations[], or on --allow-coverage-gaps). Run scripts/coverage-check.sh --scope <run-dir>/scope.json --plan <plan>. If verdict == fail, refuse the plan with PHASE=plan and REASON=uncovered required obligations: <ids> — the planner must add cases for the named obligations and re-emit. --allow-coverage-gaps downgrades the refusal to a warning (uncovered ids surfaced in QA_REPORT.md notes). Fold COVERAGE_REQUIRED / COVERAGE_COVERED / COVERAGE_UNCOVERED into the run envelope.
Launch service via scripts/launch-entry-point.sh (skipped on --no-launch or --service <ext-url>).
Health wait via scripts/health-wait.sh.
Classify cases per references/classification-rules.md (simple vs complex).
Compute dispatch waves. Pipe plan JSON through scripts/lib/topo-order.sh. Refuse plan on cycle (exit 2) or unknown dependency id (exit 1). Each wave dispatches in parallel (capped at --max-bg); the next wave starts when every case in the current wave reaches terminal state. Exception — serial agy: if X_QA_SIMPLE_RUNNER is the agy-agent runner, the simple cases in a wave are dispatched one at a time (effective --max-bg 1 for the agy runner — concurrent agy processes hang); see "Serial-agy rule" under Capability Routing. Non-agy simple runners and the complex runner keep the parallel --max-bg fanout. Cases whose deps failed are marked skipped, not fail. Templates in references/case-runner-prompts.md. Each case's full precondition chain is pre-resolved by scripts/run/resolve-preconditions.sh (depth cap 4) into X_QA_PRECONDITION_STEPS before dispatch.
Background-execution gate (CI vs local). Dispatch is run_in_background: true only when the environment looks interactive ([[ -z "$CI" && -z "$GITHUB_ACTIONS" && -z "$BUILDKITE" && -z "$GITLAB_CI" ]]). In CI, dispatch synchronously so the runner's stdout/stderr land in the build log and the run does not complete before agents finish. The --no-bg flag forces synchronous dispatch unconditionally.
Collect every dispatch terminal state (mandatory per ~/.claude/rules/background-agents.md). Never background_cancel(all=true) before collection.
Retry flaky inline up to --retry-flaky.
13.5. Exploratory bug-hunt (team) — default on a local run; skipped in CI; --no-explore opts out, --explore forces it. Gate on the Phase-11 CI predicate. Requires the service to be up (skip if Phase 8 was skipped). When scope.json.obligations[] is present, partition it into ≤6 clusters (scripts/explore/cluster-partition.sh --max-workers 6); otherwise cluster by reachable endpoints (skip if neither). Dispatch one Exploratory Worker per cluster (references/explorer-prompts.md) via X_QA_EXPLORE_MODE (native team + shared bug-board, or background fanout — references/explore-team.md), each bounded to a ≤15-probe budget, writing to <run-dir>/explore/board.jsonl. Then dedup by signature (scripts/explore/finding-merge.sh), triage each unique finding independently (references/triage-verify.md), and mint a red repro stub per confirmed finding (scripts/explore/finding-to-case.sh) for the x-bugfix route + the report. Do NOT KB-promote these — the KB is the green corpus and Phase 16 auto-promotes only green cases; a repro stub becomes a regression case only after the fix lands and it goes green (via the existing auto-promote path). Count EXPLORE_CONFIRMED from the triaged set (this step), NOT from finding-merge.sh output (which runs pre-triage, when every finding is still suspected). Fold EXPLORE_* counters into the envelope.
Teardown via launch entry's launch.teardown (skipped if Phase 8 was skipped).
Aggregate via scripts/aggregate-results.sh → QA_REPORT.md. Propagate scope.json.open_questions into the report's notes section. KB write-back (skipped on --no-kb): update kb/baselines/<endpoint>.json for every case, append a run summary line to kb/.ledger.jsonl, compute drift signals.
KB auto-promote (skipped on --no-kb or --kb-disable-auto-promote). Invoke scripts/kb-promote.sh --auto. Emits KB_PROMOTED= / KB_DEMOTED= for the envelope.
Emit envelope.
Real-QA Contract (MANDATORY)
run tests the system the way a QA engineer drives it —
it never executes the repository's own test suites. The runner MUST NOT
invoke npm test,
npm run test:e2e, pytest, playwright test, cypress run, go test,
vitest, or any project test command. Instead it drives the actual channel:
issue real requests (curl for http), adjust fixture/mock data, and mint cases
from QA_MEMORY.md + the KB corpus. launch.command starts the service only
(references/service-launch.md); it is never a test command. This holds across
every driver.
Eval class (v1). LLM/agentic features are tested with eval scorers (llm-rubric,
semantic-similarity) — x-qa scores outputs itself via the native judge-runner
(scripts/evals/score-case.sh); it MUST NOT shell out to external eval frameworks
(deepeval, promptfoo) any more than it runs the repo's test suites. The judge model
SHOULD differ from the model behind the system-under-test (operator responsibility — not
machine-enforced in v1; x-qa cannot infer the SUT's model). A judge may set QA_VERDICT=fail
only when validated against a human gold set (κ ≥ 0.90); otherwise it is advisory (warn). See
references/eval-scorers.md.
After This Skill
If invoked by x-team: emit envelope only, x-team consumes via Skill return value.
If invoked by user: print envelope + path to QA_REPORT.md + summary table.
On fail: surface offer to route into /x-skills:x-bugfix with the failed cases as input.
Confirmed exploratory findings (EXPLORE_CONFIRMED > 0) are also offered to /x-skills:x-bugfix, each carrying its minted case as the reproduction.
Persist test pattern (only when mcp.basic_memory pinned): apply the § Memory Reflex durability gate first (a reusable test pattern or real flake root cause, not a routine run summary), then one mcp__basic-memory__write_note({ title: "<pattern-kind>: <one-line>", directory: "lessons/<project-slug>", content: "<test pattern or flake observation>", tags: ["<project-slug>", "x-qa", "<framework>", "test-plan"] }) call (durability gate, placement, and Update over duplicate — same-kind only — per ../x-shared/mcp-toolbox.md § Memory Reflex / § Consumer rules). Skip silently when not pinned.
