| name | qa-test |
| description | Run an automated QA flow against a CLI or a locally-served app and report step-level pass/fail with evidence. Use when asked to QA-test a feature, exercise a flow end-to-end, or smoke-test what a PR changed. |
| version | 1.0.0 |
| tags | ["github","qa","testing","evidence"] |
QA Test
Drive a target through a sequence of steps and report step-level pass/fail
with evidence for each step. The deliverable is a QA report, not a single
verdict — partial coverage with documented failures is the expected output.
This skill uses the building skill for installing dependencies and running
the repo's build/run commands in the sandbox.
Scope (text-evidence path). In the default text phase the sandbox agent
has bash, file read, and the github tools — no browser, no screenshots. QA a
CLI directly, or a web service by starting its dev-server in the
background and exercising it with curl (status codes, response bodies,
headers). Capture stdout/stderr, exit codes, and log excerpts as per-step
evidence. Driving a real browser UI and attaching screenshots is the separate
browser-qa skill, staged only in the gated browser phase running on the
docker QA image (Playwright + Chromium). If that skill isn't in your
catalogue, mark a step that genuinely needs rendered-UI interaction
BLOCKED with the reason rather than faking a result — the browser phase
will pick it up when available.
Parse the target
The Context block gives you what to test and (usually) a PR/issue. The target
may be:
- A CLI command (
my-cli --flag, a subcommand) → run it directly.
- A local web service (an app the repo serves on a port) → start it, hit it
with
curl.
- A PR reference + focus area → read the diff and infer the flow that
exercises what changed.
- A free-text flow ("create a project, then list it") → infer concrete steps.
When a PR/issue is given, read its description + diff (from the local checkout —
see the building skill) to decide what's worth testing.
Define the steps
If the request specifies steps, use them. Otherwise design a reasonable flow:
- CLI: invoke with valid input → assert output/exit code → invoke an edge
case (bad flag, missing arg) → assert it's handled → done.
- Web service: start the server → wait until it's ready → hit the primary
endpoint(s) → assert status + body → hit an error case → assert handling →
stop the server.
State the steps and their success criteria before running. If the flow or the
success criteria are genuinely ambiguous and you can't infer them from the diff,
say what's unclear in the report rather than inventing a pass.
Run each step
Follow the building skill to install and build first. Then run each step in
order, capturing evidence (command + stdout/stderr + exit code) at every step.
- Record each step's result as you go — PASS, FAIL, or BLOCKED.
- On failure, continue to the next step for maximum coverage — unless the
failure blocks everything downstream (e.g. the server never starts, or a login
step fails). If it's a hard blocker, record it, mark the rest BLOCKED, and stop.
- Treat a wrong result as a real finding, not your mistake. Don't edit the code
under test to make a step pass.
Report
Produce your report as your final message in this shape — the workflow posts
it for you (don't github_add_issue_comment yourself; that would double-post).
Keep it concise: a one-line environment summary, the results table (terse
Evidence cells), the failures, and a short coverage note — not a step-by-step
narration.
## QA Test: <target>
**Environment:** <branch / commit, package manager, how the app was run>
### Results
| Step | Status | Evidence |
|------|--------|----------|
| <step description> | PASS / FAIL / BLOCKED | <command + key output / exit code> |
### Issues found
- <each FAIL with the expected vs observed behaviour and the evidence>
### Coverage
<what was tested, and what was not tested and why (e.g. a UI-only step that the
text-evidence sandbox can't drive).>
Every defined step must have a row and a result. Never report a flow as
"passed" with steps you didn't actually run — list them as untested instead.
