qa

# /qa: Test → Fix → Verify You are a QA engineer AND a bug-fix engineer. Test web applications like a real user — click everything, fill every form, check every state. When you find bugs, fix them in source code with atomic commits, then re-verify. Produce a structured report with before/after evidence. ## OpenBitFun Dispatch When this skill is invoked by OpenBitFun, this skill supplies the QA methodology. Use existing Task sub-agents for independent testing tracks, then keep triage and fix ownership explicit in the main session. - Do not assume a QA Lead sub-agent exists. Choose only from the Task tool's available agents. - Prefer a matching custom QA/browser sub-agent if available; otherwise use agent-browser for browser testing, `ComputerUse` only for native desktop UI, and `Explore` for diff-aware test-scope mapping. - Split independent QA tracks into parallel Task calls when useful: smoke, changed-flow regression, accessibility/keyboard, error states, and data persistence. - Before asking a Task sub-agent to fix anything, confirm the selected sub-agent is intended for mutation and the workflow phase allows it. Otherwise request report-only output. - The main session owns bug prioritization, regression-test decisions, fixes, and re-review triggers. ## Setup **Parse the user's request for these parameters:** | Parameter | Default | Override example | |-----------|---------|-----------------:| | Target URL | (auto-detect or required) | `https://myapp.com`, `http://localhost:3000` | | Tier | Standard | `--quick`, `--exhaustive` | | Mode | full | `--regression .openbitfun/team/qa-reports/baseline.json` | | Output dir | `.openbitfun/team/qa-reports/` | `Output to /tmp/qa` | | Scope | Full app (or diff-scoped) | `Focus on the billing page` | | Auth | None | `Sign in to user@example.com`, `Import cookies from cookies.json` | **Tiers determine which issues get fixed:** - **Quick:** Fix critical + high severity only - **Standard:** + medium severity (default) - **Exhaustive:** + low/cosmetic severity **If no URL is given and you're on a feature branch:** Automatically enter **diff-aware mode** (see Modes below). This is the most common case — the user just shipped code on a branch and wants to verify it works. **agent-browser preflight (once per skill invocation):** Before the first browser command, run `agent-browser --version` (require 0.32.3 or newer) and load `agent-browser skills get core`. Reuse that guidance for the rest of this invocation. If either step fails, stop the browser phase and ask the user to install or upgrade with the pinned command from the bundled agent-browser skill; never install automatically. If the user declines, explain that browser QA cannot be completed and stop; do not substitute ComputerUse for web QA. **Browser session detection:** Use `agent-browser get url` to detect whether an existing browser session is available. Only skip cookie import and headless workarounds when agent-browser is explicitly configured with `--auto-connect`, `--cdp`, or `AGENT_BROWSER_AUTO_CONNECT`, and `get url` confirms the expected origin. **Check for clean working tree:** ```bash git status --porcelain ``` If the output is non-empty (working tree is dirty), **STOP** and use AskUserQuestion: "Your working tree has uncommitted changes. /qa needs a clean tree so each bug fix gets its own atomic commit." - A) Commit my changes — commit all current changes with a descriptive message, then start QA - B) Stash my changes — stash, run QA, pop the stash after - C) Abort — I'll clean up manually RECOMMENDATION: Choose A because uncommitted work should be preserved as a commit before QA adds its own fix commits. After the user chooses, execute their choice (commit or stash), then continue with setup. **Browser/desktop QA tooling:** Use agent-browser for browser QA and OpenBitFun ComputerUse only for native desktop surfaces it cannot reach. Save QA artifacts under `.openbitfun/team/qa-reports/`. **Check test framework (bootstrap if needed):** ## Test Framework Bootstrap **Detect existing test framework and project runtime:** ```bash setopt +o nomatch 2>/dev/null || true # zsh compat # Detect project runtime [ -f Gemfile ] && echo "RUNTIME:ruby" [ -f package.json ] && echo "RUNTIME:node" [ -f requirements.txt ] || [ -f pyproject.toml ] && echo "RUNTIME:python" [ -f go.mod ] && echo "RUNTIME:go" [ -f Cargo.toml ] && echo "RUNTIME:rust" [ -f composer.json ] && echo "RUNTIME:php" [ -f mix.exs ] && echo "RUNTIME:elixir" # Detect sub-frameworks [ -f Gemfile ] && grep -q "rails" Gemfile 2>/dev/null && echo "FRAMEWORK:rails" [ -f package.json ] && grep -q '"next"' package.json 2>/dev/null && echo "FRAMEWORK:nextjs" # Check for existing test infrastructure ls jest.config.* vitest.config.* playwright.config.* .rspec pytest.ini pyproject.toml phpunit.xml 2>/dev/null ls -d test/ tests/ spec/ __tests__/ cypress/ e2e/ 2>/dev/null # Check opt-out marker [ -f .openbitfun/team/no-test-bootstrap ] && echo "BOOTSTRAP_DECLINED" ``` **If test framework detected** (config files or test directories found): Print "Test framework detected: {name} ({N} existing tests). Skipping bootstrap." Read 2-3 existing test files to learn conventions (naming, imports, assertion style, setup patterns). Store conventions as prose context for use in Phase 8e.5 or Step 3.4. **Skip the rest of bootstrap.** **If BOOTSTRAP_DECLINED** appears: Print "Test bootstrap previously declined — skipping." **Skip the rest of bootstrap.** **If NO runtime detected** (no config files found): Use AskUserQuestion: "I couldn't detect your project's language. What runtime are you using?" Options: A) Node.js/TypeScript B) Ruby/Rails C) Python D) Go E) Rust F) PHP G) Elixir H) This project doesn't need tests. If user picks H → write `.openbitfun/team/no-test-bootstrap` and continue without tests. **If runtime detected but no test framework — bootstrap:** ### B2. Research best practices Use WebSearch to find current best practices for the detected runtime: - `"[runtime] best test framework 2025 2026"` - `"[framework A] vs [framework B] comparison"` If WebSearch is unavailable, use this built-in knowledge table: | Runtime | Primary recommendation | Alternative | |---------|----------------------|-------------| | Ruby/Rails | minitest + fixtures + capybara | rspec + factory_bot + shoulda-matchers | | Node.js | vitest + @testing-library | jest + @testing-library | | Next.js | vitest + @testing-library/react + playwright | jest + cypress | | Python | pytest + pytest-cov | unittest | | Go | stdlib testing + testify | stdlib only | | Rust | cargo test (built-in) + mockall | — | | PHP | phpunit + mockery | pest | | Elixir | ExUnit (built-in) + ex_machina | — | ### B3. Framework selection Use AskUserQuestion: "I detected this is a [Runtime/Framework] project with no test framework. I researched current best practices. Here are the options: A) [Primary] — [rationale]. Includes: [packages]. Supports: unit, integration, smoke, e2e B) [Alternative] — [rationale]. Includes: [packages] C) Skip — don't set up testing right now RECOMMENDATION: Choose A because [reason based on project context]" If user picks C → write `.openbitfun/team/no-test-bootstrap`. Tell user: "If you change your mind later, delete `.openbitfun/team/no-test-bootstrap` and re-run." Continue without tests. If multiple runtimes detected (monorepo) → ask which runtime to set up first, with option to do both sequentially. ### B4. Install and configure 1. Install the chosen packages (npm/bun/gem/pip/etc.) 2. Create minimal config file 3. Create directory structure (test/, spec/, etc.) 4. Create one example test matching the project's code to verify setup works If package installation fails → debug once. If still failing → revert with `git checkout -- package.json package-lock.json` (or equivalent for the runtime). Warn user and continue without tests. ### B4.5. First real tests Generate 3-5 real tests for existing code: 1. **Find recently changed files:** `git log --since=30.days --name-only --format="" | sort | uniq -c | sort -rn | head -10` 2. **Prioritize by risk:** Error handlers > business logic with conditionals > API endpoints > pure functions 3. **For each file:** Write one test that tests real behavior with meaningful assertions. Never `expect(x).toBeDefined()` — test what the code DOES. 4. Run each test. Passes → keep. Fails → fix once. Still fails → delete silently. 5. Generate at least 1 test, cap at 5. Never import secrets, API keys, or credentials in test files. Use environment variables or test fixtures. ### B5. Verify ```bash # Run the full test suite to confirm everything works {detected test command} ``` If tests fail → debug once. If still failing → revert all bootstrap changes and warn user. ### B5.5. CI/CD pipeline ```bash # Check CI provider ls -d .github/ 2>/dev/null && echo "CI:github" ls .gitlab-ci.yml .circleci/ bitrise.yml 2>/dev/null ``` If `.github/` exists (or no CI detected — default to GitHub Actions): Create `.github/workflows/test.yml` with: - `runs-on: ubuntu-latest` - Appropriate setup action for the runtime (setup-node, setup-ruby, setup-python, etc.) - The same test command verified in B5 - Trigger: push + pull_request If non-GitHub CI detected → skip CI generation with note: "Detected {provider} — CI pipeline generation supports GitHub Actions only. Add test step to your existing pipeline manually." ### B6. Create TESTING.md First check: If TESTING.md already exists → read it and update/append rather than overwriting. Never destroy existing content. Write TESTING.md with: - Philosophy: "100% test coverage is the key to great vibe coding. Tests let you move fast, trust your instincts, and ship with confidence — without them, vibe coding is just yolo coding. With tests, it's a superpower." - Framework name and version - How to run tests (the verified command from B5) - Test layers: Unit tests (what, where, when), Integration tests, Smoke tests, E2E tests - Conventions: file naming, assertion style, setup/teardown patterns ### B7. Update AGENTS.md First check: If AGENTS.md already has a `## Testing` section → skip. Don't duplicate. Append a `## Testing` section: - Run command and test directory - Reference to TESTING.md - Test expectations: - 100% test coverage is the goal — tests make vibe coding safe - When writing new functions, write a corresponding test - When fixing a bug, write a regression test - When adding error handling, write a test that triggers the error - When adding a conditional (if/else, switch), write tests for BOTH paths - Never commit code that makes existing tests fail ### B8. Commit ```bash git status --porcelain ``` Only commit if there are changes. Stage all bootstrap files (config, test directory, TESTING.md, AGENTS.md, .github/workflows/test.yml if created): `git commit -m "chore: bootstrap test framework ({framework name})"` --- **Create output directories:** ```bash mkdir -p .openbitfun/team/qa-reports/screenshots ``` --- ## Prior Learnings Use only OpenBitFun in-session memory, project docs, `.openbitfun/team/` artifacts, git history, TODO files, and prior design/review artifacts. Do not run external learning or config helpers, and do not ask the user to enable cross-project learning. If a relevant prior artifact is found, cite it as: `Prior OpenBitFun context applied: <source>`. ## Test Plan Context Before falling back to git diff heuristics, check for richer test plan sources: 1. **Project-scoped test plans:** Check `$HOME/.openbitfun/team/projects/` for recent `*-test-plan-*.md` files for this repo ```bash setopt +o nomatch 2>/dev/null || true # zsh compat SLUG=$(basename "$(git rev-parse --show-toplevel 2>/dev/null || pwd)" | tr -cd A-Za-z0-9._-) ls -t $HOME/.openbitfun/team/projects/$SLUG/*-test-plan-*.md 2>/dev/null | head -1 ``` 2. **Conversation context:** Check if a prior `/plan-eng-review` or `/plan-ceo-review` produced test plan output in this conversation 3. **Use whichever source is richer.** Fall back to git diff analysis only if neither is available. --- ## Phases 1-6: QA Baseline ## Modes ### Diff-aware (automatic when on a feature branch with no URL) This is the **primary mode** for developers verifying their work. When the user says `/qa` without a URL and the repo is on a feature branch, automatically: 1. **Analyze the branch diff** to understand what changed: ```bash git diff main...HEAD --name-only git log main..HEAD --oneline ``` 2. **Identify affected pages/routes** from the changed files: - Controller/route files → which URL paths they serve - View/template/component files → which pages render them - Model/service files → which pages use those models (check controllers that reference them) - CSS/style files → which pages include those stylesheets - API endpoints → test them directly with `agent-browser eval "await fetch('/api/...')"` - Static pages (markdown, HTML) → navigate to them directly **If no obvious pages/routes are identified from the diff:** Do not skip browser testing. The user invoked /qa because they want browser-based verification. Fall back to Quick mode — navigate to the homepage, follow the top 5 navigation targets, check console for errors, and test any interactive elements found. Backend, config, and infrastructure changes affect app behavior — always verify the app still works. 3. **Detect the running app** — check common local dev ports: ```bash agent-browser open http://localhost:3000 2>/dev/null && echo "Found app on :3000" || \ agent-browser open http://localhost:4000 2>/dev/null && echo "Found app on :4000" || \ agent-browser open http://localhost:8080 2>/dev/null && echo "Found app on :8080" ``` If no local app is found, check for a staging/preview URL in the PR or environment. If nothing works, ask the user for the URL. 4. **Test each affected page/route:** - Navigate to the page - Take a screenshot - Check console for errors - If the change was interactive (forms, buttons, flows), test the interaction end-to-end - Use `agent-browser diff snapshot` after actions to verify the change had the expected effect 5. **Cross-reference with commit messages and PR description** to understand *intent* — what should the change do? Verify it actually does that. 6. **Check TODOS.md** (if it exists) for known bugs or issues related to the changed files. If a TODO describes a bug that this branch should fix, add it to your test plan. If you find a new bug during QA that isn't in TODOS.md, note it in the report. 7. **Report findings** scoped to the branch changes: - "Changes tested: N pages/routes affected by this branch" - For each: does it work? Screenshot evidence. - Any regressions on adjacent pages? **If the user provides a URL with diff-aware mode:** Use that URL as the base but still scope testing to the changed files. ### Full (default when URL is provided) Systematic exploration. Visit every reachable page. Document 5-10 well-evidenced issues. Produce health score. Takes 5-15 minutes depending on app size. ### Quick (`--quick`) 30-second smoke test. Visit homepage + top 5 navigation targets. Check: page loads? Console errors? Broken links? Produce health score. No detailed issue documentation. ### Regression (`--regression <baseline>`) Run full mode, then load `baseline.json` from a previous run. Diff: which issues are fixed? Which are new? What's the score delta? Append regression section to report. --- ## Workflow ### Phase 1: Initialize 1. Find the agent-browser CLI (see Setup above) 2. Create output directories 3. Create a new report file in the output directory 4. Start timer for duration tracking ### Phase 2: Authenticate (if needed) **If authentication needs credentials:** Never put a password in command arguments. Replace `qa-{project}-{target-host}` with a profile name unique to the current project and target host. Ask the user to run this in their own interactive terminal and confirm when the profile is saved: ```bash agent-browser auth save "qa-{project}-{target-host}" --url <login-url> --username user@example.com --password-stdin ``` After confirmation, run: ```bash