qa-test

name

qa-test

description

Use after a deployment to run QA golden-path tests (core lanes + LLM-driven UI exploration) and classify failures. Triggers on /qa-test, "qa test", "deploy sonrası test", "qa agent çalıştır", "post-deploy qa", "run qa tests", "deployment test et", "qa koş". Runs the deterministic core (smoke/backend/ui-regression/bot lanes) then drives Claude-in-Chrome for critical UI flows, maps every failure to an apex-debug symptom class, and prints a pass/fail summary with handoff suggestions.

QA Test Agent Skill

Run a full QA sweep after a deployment: deterministic lanes (HTTP smoke, in-pod backend tests, Playwright regression, optional bot live) then LLM-driven UI exploration for critical flows and new-feature pages. Classify failures → apex-debug handoff.

Design spec: docs/superpowers/specs/2026-06-14-qa-test-agent-design.md Core CLI: qa_agent/ at repo root (python -m qa_agent run) Manifest schema: qa_agent/README.md

Hard boundaries

• No deploy / no mutations. This skill only READS the deployed environment and drives a browser. No kubectl apply, no DB writes, no git push.
• UI-exploration results are non-deterministic (LLM/visual). They go in a SEPARATE section from deterministic lane results — visual diffs do NOT become hard fails. (R3 guard — see §5.)
• Honest reachability (R4 guard). If the manifest base_url is an internal DC IP (10.185.63.33) and it is not reachable from this machine, report "unreachable from this host" and do NOT silently pass UI-exploration. (See §4.)
• Self-signed TLS. The k3s_dc QA environment uses self-signed certificates. The browser MUST accept them (Playwright ignoreHTTPSErrors: true; httpx verify=False). Never treat a cert warning as a test failure.

---

SOP — Run in this order

0. Intake

Ask (or infer from context):

1. Which env? (dev | qa | test) — determines --env flag and access method.
2. Which manifest? Default: qa-agent.yaml at repo root. If absent, ask the operator to provide the path.
3. Scope override? Default: all lanes. Operator may say --only smoke.

If the operator does not provide env, ask before running — the access path depends on it.

---

1. Locate + read the manifest

# From repo root (e.g. /Users/maydogan/Documents/apex_folders/self_apex)
cat qa-agent.yaml

Parse these fields (used in later steps):

• environments.<env>.base_url — the URL to test against
• ui_exploration.critical_flows — list of flows for lane 4
• ui_exploration.enabled — whether lane 4 should run at all
• report.output_dir — where JSON+markdown reports are written

---

2. Reachability check (R4 guard — BEFORE running anything)

# Quick TCP check; adjust port from base_url
curl -sk --max-time 5 -o /dev/null -w "%{http_code}" <base_url>/health || echo "UNREACHABLE"

If unreachable:

• Mark Lane 4 (UI-exploration) as UNREACHABLE FROM THIS HOST — do NOT mark it pass.
• Continue with lanes 1–3/5 only IF the core can reach the target (it uses the same base_url). If even smoke fails to connect, abort and report connectivity issue.
• Instruct the operator: "Run /qa-test from a machine on the DC network (k3s_dc SSH jump) or use the CyberArk bastion to reach 10.185.63.33:8443."

If reachable: proceed normally.

---

3. Run the deterministic core (lanes 1–3 and optionally 5)

cd /path/to/repo   # e.g. /Users/maydogan/Documents/apex_folders/self_apex
python -m qa_agent run --manifest qa-agent.yaml --env <env>

Optional overrides:

• --only smoke — smoke only (fastest, use when you just want a liveness check)
• --ci — smoke=blocking; other lanes=report-only (mirrors CI behavior)

Capture the output and note:

• Per-lane status: PASS / FAIL / SKIP / ERROR
• The report file paths printed at the end (JSON + markdown under report.output_dir)
• Any [Lane N] block with FAIL — carry these into §5 classification

Read the written JSON report for the machine-readable detail:

cat docs/qa-runs/<timestamp>-<env>.json

---

4. Lane 4 — UI exploration via Claude-in-Chrome

Only run if ui_exploration.enabled: true in the manifest AND the base_url is reachable (R4 guard, §2). If skipped, record reason explicitly.

For each flow in ui_exploration.critical_flows:

4a. Prepare the browser

Use the tabs_context_mcp tool (or ToolSearch: mcp__tabs_context_mcp__*) to open a new tab at the environment base_url. Accept the self-signed cert prompt if shown.

Tool: mcp__tabs_context_mcp__tabs_create_tab  (or equivalent from ToolSearch)
url: <base_url>    # e.g. https://10.185.63.33:8443

Deferred-tool pattern (from CLAUDE.md): If the browser MCP tools are not loaded in this session, use ToolSearch to locate them before invoking. Do not assume the tool name — look it up first.

4b. Execute each critical flow

For every flow entry in ui_exploration.critical_flows:

1. Open the target URL (base_url + flow start path).
2. Execute the steps listed in the flow entry in order (fill inputs, click buttons).
3. Assert the expect condition (visible element, no error modal, correct URL change).
4. Take a screenshot — save to docs/screenshots/qa-<flow-slug>-<timestamp>.png.
5. Check the browser console for errors (JavaScript errors, failed network requests).
6. Check the network tab for any 4xx/5xx responses during the flow.

Record per flow:

• status: pass | needs-review | fail
• screenshot_path
• console_errors: list of any JS errors seen
• network_errors: list of any HTTP error responses (status + URL)
• notes: what was observed

4c. R3 guard — separate UI-exploration results

UI-exploration results are LLM-driven and non-deterministic. Report them in a distinct section (§ "UI Exploration (needs-review, non-deterministic)") — never mixed into the deterministic lane table. A needs-review status never causes the overall run to be marked FAIL.

---

5. Classify every failure

For each failing or erroring lane from the deterministic core AND for each UI-exploration flow with needs-review or fail, map it to an apex-debug symptom class using this table (mirrors qa_agent/report.py :: classify_failure):

Condition	Symptom Class
Lane = bot (any failure)	Class 1 — Slack-silent (bot unresponsive / service down)
Details contain: connect, connection, unreachable, timeout (any lane)	Class 1 — Slack-silent (service down)
Details contain: mcp, tool call, mcp-fail	Class 2 — MCP-tool failure
Details contain: routing, route, specialist, wrong lane	Class 4 — Routing / wrong specialist
Details contain: 500, 502, 503, 5xx, internal server (smoke or backend)	Class 3 — API 5xx / backend error
Lane = ui or playwright or UI-exploration failure	Class UI — Frontend rendering / UI regression
Default (none of the above match)	Class 3 — API 5xx / backend error

For each failure, note:

• The apex-debug class label
• The specific failure detail (what the lane reported)
• The recommended next step

---

6. Print the summary report

Output this in your chat reply:

---

QA Run: <project> — env=<env> — <timestamp>

Deterministic Lanes

Lane	Status	Duration	Details
smoke	PASS/FAIL	Xs	...
backend	PASS/FAIL	Xs	...
ui-regression	PASS/FAIL	Xs	...
bot	PASS/FAIL	Xs	...

Overall (deterministic): PASS / FAIL

UI Exploration — needs-review (non-deterministic, LLM-driven)

Flow	Status	Screenshot	Console Errors	Network Errors
Admin login → dashboard	needs-review / pass	docs/screenshots/qa-...png	none	none
...

Note: UI exploration findings are non-deterministic. Review screenshots before acting on needs-review results.

Failure Classification + Handoff

Lane / Flow	Failure detail	Symptom Class	Action
smoke	Connection refused	Class 1 — service down	Check pod status → run /apex-debug env=qa
ui-exploration: Create bot	JS error on submit	Class UI — frontend rendering	Review screenshot → fix JS error

Tested fingerprint: <image_ref> @ <image_digest> (<timestamp_utc>) (Reliability: reliable / UNRELIABLE — <unreliable_reason>)

Reports written:

• JSON: docs/qa-runs/<timestamp>-<env>.json
• Markdown: docs/qa-runs/<timestamp>-<env>.md

---

7. Handoff

For any FAIL or ERROR in deterministic lanes:

• Say: "→ Run /apex-debug env=<env> and paste the failure details above."

For any needs-review in UI exploration:

• Say: "→ Review the screenshot at docs/screenshots/qa-<flow>-<timestamp>.png. If confirmed as a bug, open an issue and optionally run /apex-debug for root cause."

For overall PASS (all deterministic lanes green, UI exploration at most needs-review):

• Say: "QA sweep complete. All deterministic lanes PASS. UI exploration needs human review of the screenshots above."

---

Quick reference

CLI flags

Flag	Description
--manifest	Path to qa-agent.yaml (default: qa-agent.yaml)
--env	Environment name (dev, qa, test)
--only smoke	Run smoke lane only (fast liveness check)
--ci	CI mode: smoke=blocking, others=report-only

Symptom class → apex-debug mapping

Class	Label	First step
1	Slack-silent / service down	Check pod status, ARQ queue, Slack ack path
2	MCP-tool failure	Check MCP server health, OAuth token state
3	API 5xx / backend error	Check backend logs, 500 trace_spans
4	Routing / wrong specialist	Check orchestrator catalog, router decision log
UI	Frontend rendering / regression	Check browser console, React errors, screenshot

Access method by env

env	How to reach	Browser
dev	https://localhost	local Chrome
qa (k3s_dc)	https://10.185.63.33:8443 — internal DC, CyberArk bastion required from outside	local Chrome (operator machine on DC network)
test (AWS k4s)	apex-test.getirfinance.com	local Chrome

k3s_dc reachability: 10.185.63.33:8443 is internal to the GetirFinans DC. If running from outside (Claude Code Bash sandbox or operator machine not on VPN/DC network), the R4 guard will catch the connectivity failure and report it honestly. Do NOT silently mark UI-exploration as pass when the environment is unreachable.

Related skills

• /apex-debug — root-cause diagnosis for APEX malfunctions; this skill hands off to it.
• /apex-live-tests — bot-level live tests (mock ARQ enqueue, Slack golden-path); use alongside or instead of lane 5 when focused bot coverage is needed.

Agent flow summary

1. Read manifest → parse base_url + ui_exploration.critical_flows
2. R4 reachability check (curl base_url) → gate UI-exploration
3. python -m qa_agent run --manifest qa-agent.yaml --env <env>
   → lanes 1-3 (+ 5 if enabled) → JSON report
4. For each flow in ui_exploration.critical_flows (if reachable):
   → Claude-in-Chrome: open URL, execute steps, screenshot, check console/network
   → R3 guard: results go to SEPARATE non-deterministic section
5. Classify all failures: lane fail → symptom class (Class 1/2/3/4/UI)
6. Print summary table (deterministic | UI-exploration | classification + handoff)
7. Suggest /apex-debug for any hard failures