qa-verify

# /qa-verify Capture before/after visual evidence of PR changes using Playwright MCP and post a comparison to the GitHub PR. ## Current State Branch: !`git rev-parse --abbrev-ref HEAD` PR: !`gh pr view $pr --json number,title,body,headRefName,baseRefName 2>/dev/null || echo "No PR found"` Base branch: !`gh pr view $pr --json baseRefName -q '.baseRefName' 2>/dev/null || echo "main"` Commits since base: !`git log "$(gh pr view $pr --json baseRefName -q '.baseRefName' 2>/dev/null || echo main)..HEAD" --oneline --no-merges 2>/dev/null || echo "No commits ahead of base"` Changed files: !`git diff "$(gh pr view $pr --json baseRefName -q '.baseRefName' 2>/dev/null || echo main)...HEAD" --stat 2>/dev/null || echo "Unable to diff"` ## Phase 0: Prerequisites ```bash bash "${CLAUDE_SKILL_DIR}/scripts/check-prerequisites.sh" ``` If any tools are missing, the script outputs an `INSTALL_COMMANDS` section with platform-specific install commands (brew, dnf, or apt). Use `AskUserQuestion` to offer to run them. If the user agrees, run each install command, then re-run the prerequisites check to confirm everything passes. Verify Playwright MCP is available by calling `browser_snapshot`. If unavailable, the user needs to configure it **before starting this session** — adding MCP servers requires a session restart. Configure with `--ignore-https-errors` and `--save-video` for video recording: ```json { "mcpServers": { "playwright": { "command": "npx", "args": ["@playwright/mcp@latest", "--ignore-https-errors", "--save-video=1920x1080"] } } } ``` The `--save-video=1920x1080` flag records a video of the entire browser session from page creation to `browser_close`. Videos are saved as `.webm` files automatically. ## Phase 1: Gather Context Use the injected state above. Additionally: Extract the Jira key from the PR title or branch name (`CONSOLE-\d+` or `OCPBUGS-\d+`). If Atlassian MCP is available, call `getJiraIssue` with `cloudId: "redhat.atlassian.net"`. From the PR body, parse: - **Test setup** section (between `**Test setup:**` and the next `**` header) - **Test cases** section (between `**Test cases:**` and the next `**` header) - **Solution description** section ## Phase 2: Verification Plan Derive verification steps and present to the user with `AskUserQuestion`. Wait for confirmation. Sources (use what's available): 1. PR test cases → each becomes a verification step with route, action, expected result 2. Jira ticket → bug repro steps or acceptance criteria map to pages 3. Diff → changed components map to affected routes 4. Fallback → dashboard and pages related to changed files 5. **Backend-only changes** — if the diff only touches `pkg/`, `cmd/`, or `*.go` files: - Check whether the fix depends on cluster resources (CRDs, operators) that may not exist in the local dev environment. If so, note this limitation in the plan. - Offer the user a choice: (a) run `go test ./pkg/...` on both branches and compare results instead of the full build-and-screenshot flow, or (b) proceed with UI verification but focus on functional behavior (button states, API responses) rather than pixel comparison. - If the user chooses test comparison, run the tests, diff the results, and skip Phases 3-4. **Route guidelines:** - Use namespace-scoped routes (`/k8s/ns/default/...`) instead of `/k8s/all-namespaces/...` for any page with forms or project selectors. With all-namespaces, no project is selected, so form validation independently disables submit buttons — making verification useless. - Pick a namespace that exists on the cluster (e.g., `default`, `openshift-console`). Present as a table: | # | Route | Action | Expected Result | |---|-------|--------|-----------------| | 1 | /dashboards | Navigate, wait load | Dashboard renders | | 2 | /k8s/ns/default/pods | Click a pod | Pod detail page opens | | 3 | /k8s/ns/openshift-console/deployments/console/environment | View env tab, click Add from ConfigMap | Value-from-pair form opens | The `$pr` argument (from `/qa-verify <url-or-number>`) is expanded by skill substitution into the content before you see it. If provided, it is used for `gh pr view` and `gh pr checkout`. It may be a full GitHub URL or just a number. **PR suitability:** This skill works best for PRs with visible UI changes (new components, layout changes, styling) where before/after screenshots show a clear diff. For logic-only PRs (state management, hooks, refactors), the skill still provides value as a regression check — identical screenshots confirm the refactor didn't break anything visually. In these cases, also check for console errors via `browser_console_messages` after each navigation step to catch runtime regressions that don't produce visible changes. **Data-loading pages:** Pages like `/k8s/cluster/projects` load data via WebSocket — the page may render before data arrives. Use `browser_wait_for` with `time: 5` (longer) for data-heavy pages, or poll for a specific element via `browser_evaluate`. For paginated lists, append `?perPage=200` to show all items, or use the filter/search input to find specific items instead of scrolling through pages. **Verification depth:** Err on the side of comprehensive testing over simplicity. Think about corner cases the PR might affect — empty states, error states, loading states, boundary conditions (e.g., very long names, special characters, zero items, maximum items). If the PR touches a component, verify it in multiple contexts where it appears, not just the most obvious one. A false negative (missing a regression) is worse than spending extra time on verification. ## Phase 3: Capture Evidence Run the verification plan twice: baseline (`${BASE_BRANCH}`) first, then candidate (PR branch). ### Setup ```bash REPO_ROOT=$(git rev-parse --show-toplevel) ORIGINAL_BRANCH=$(git rev-parse --abbrev-ref HEAD) # $pr is expanded by skill substitution to the argument value (URL or number) PR_NUMBER=$(echo "$pr" | grep -oE '[0-9]+' | tail -1) if [ -z "$PR_NUMBER" ]; then PR_NUMBER=$(gh pr view --json number -q '.number' 2>/dev/null || echo "") fi PR_BRANCH=$(gh pr view --json headRefName -q '.headRefName' 2>/dev/null || echo "$ORIGINAL_BRANCH") # Derive baseline branch from PR target — release-4.x for backports, main otherwise BASE_BRANCH=$(gh pr view ${PR_NUMBER} --json baseRefName -q '.baseRefName' 2>/dev/null || echo "main") # Track whether the candidate branch already exists locally (for cleanup) PR_BRANCH_LOCAL=$(gh pr view ${PR_NUMBER} --json headRefName -q '.headRefName' 2>/dev/null || echo "") CANDIDATE_BRANCH_EXISTED=$(git rev-parse --verify "${PR_BRANCH_LOCAL}" >/dev/null 2>&1 && echo "yes" || echo "no") TIMESTAMP=$(date +%Y%m%d-%H%M%S) ARTIFACTS="${REPO_ROOT}/.artifacts/qa-verify/${TIMESTAMP}" mkdir -p "${ARTIFACTS}/baseline/screenshots" "${ARTIFACTS}/candidate/screenshots" # Evidence is posted via gh's native `--attach`. Check viability now, before doing any of the # expensive build/capture work, and stop immediately if evidence could never be posted. # # --attach needs write (push) access to the repo being commented on, which most contributors # won't have on an org repo like openshift/console. If so, fall back to STAGING: upload once # to a personal fork (which gh's --attach *can* push to) via a PR comment or scratch issue, # then reference the resulting hosted URLs in a plain (non-attach) comment on the target repo. # See stage-attachments.sh for why PRs are preferred over issues there. REPO=$(gh repo view --json nameWithOwner -q '.nameWithOwner' 2>/dev/null || echo "") USE_ATTACH="no" STAGING_REPO="" if [ -n "$REPO" ]; then CAN_PUSH=$(gh api "repos/${REPO}" --jq '.permissions.push' 2>/dev/null || echo "false") if [ "$CAN_PUSH" = "true" ]; then USE_ATTACH="yes" else # A user's fork is not guaranteed to share the upstream repo's name (e.g. a fork of # openshift/console named "john" instead of "console"), search the user's own repos # for one whose *parent* matches. /user/repos returns the simple representation (parent # is always null there), so each fork candidate needs its own fetch to check .parent.full_name. while IFS= read -r fork_candidate; do [ -z "$fork_candidate" ] && continue INFO=$(gh api "repos/${fork_candidate}" --jq '(.parent.full_name // "") + "\t" + (.permissions.push|tostring)' 2>/dev/null || echo "") PARENT="${INFO%%$'\t'*}" CAN_PUSH_FORK="${INFO##*$'\t'}" if [ "$PARENT" = "$REPO" ] && [ "$CAN_PUSH_FORK" = "true" ]; then STAGING_REPO="$fork_candidate" break fi done < <(gh api user/repos --paginate --jq '.[] | select(.fork==true) | .full_name' 2>/dev/null || true) if [ -n "$STAGING_REPO" ]; then # Staging also needs an existing PR or Issues enabled on the fork — see # stage-attachments.sh's own check for the authoritative, up-to-date version of this. HAS_PR=$(gh pr list --repo "$STAGING_REPO" --state all --limit 1 --json number -q '.[0].number' 2>/dev/null || true) HAS_ISSUES=$(gh api "repos/${STAGING_REPO}" --jq '.has_issues' 2>/dev/null || echo "false") if [ -z "$HAS_PR" ] && [ "$HAS_ISSUES" != "true" ]; then echo "STOP: found fork ${STAGING_REPO} to stage uploads through, but it has no" >&2 echo "existing pull request and Issues are disabled. Fix one of these, then retry:" >&2 echo " gh api --method PATCH repos/${STAGING_REPO} -f has_issues=true" >&2 echo " # or open any PR on ${STAGING_REPO}" >&2 exit 1 fi fi fi fi if [ "$USE_ATTACH" = "no" ] && [ -z "$STAGING_REPO" ]; then echo "STOP: no write access to ${REPO:-<unknown repo>}, and no personal fork of it found to" >&2 echo "stage uploads through. Fork the repo, then retry. No PR comment will be posted." >&2 exit 1 fi ``` Use `PR_BRANCH` (the PR's actual head ref name) in metadata, not `ORIGINAL_BRANCH` (which is whatever local branch you happen to be on). Copy skill scripts to the artifacts directory so they survive branch switches (the scripts live on the PR branch and disappear when checking out the base branch): ```bash cp -R "${CLAUDE_SKILL_DIR}/scripts" "${ARTIFACTS}/scripts" chmod +x "${ARTIFACTS}/scripts/"*.sh ``` All subsequent script calls MUST use `${ARTIFACTS}/scripts/` instead of `${CLAUDE_SKILL_DIR}/scripts/`. ### For each lane (baseline first, then candidate): #### 3.1 Switch branch **Baseline:** ```bash STASH_COUNT_BEFORE=$(git stash list 2>/dev/null | wc -l | tr -d ' ') git stash --include-untracked -m "qa-verify-stash" 2>/dev/null || true STASH_COUNT_AFTER=$(git stash list 2>/dev/null | wc -l | tr -d ' ') QA_STASH_CREATED=$([ "$STASH_COUNT_AFTER" -gt "$STASH_COUNT_BEFORE" ] && echo "yes" || echo "no") git checkout "${BASE_BRANCH}" ``` **Candidate** (after baseline is complete): use `gh pr checkout` to handle forks correctly. External contributor PRs come from forks — `git checkout <branch>` will fail because the branch doesn't exist on origin. `gh pr checkout` fetches the correct ref automatically. ```bash gh pr checkout "${PR_NUMBER}" 2>/dev/null || git checkout "${ORIGINAL_BRANCH}" CANDIDATE_CHECKOUT_SHA=$(git rev-parse HEAD) if [ "${QA_STASH_CREATED}" = "yes" ]; then git stash pop 2>/dev/null || true fi ``` #### 3.2 Build Clean and rebuild frontend + backend in parallel (use `timeout: 600000`): ```bash bash "${ARTIFACTS}/scripts/rebuild.sh" ``` #### 3.3 Start server Replace `<LANE>` with `baseline` or `candidate`: ```bash bash "${ARTIFACTS}/scripts/backend.sh" --start <LANE> ``` #### 3.4 Navigate and capture Set viewport: `browser_resize` → width: 1920, height: 1080 For each verification step: 1. `browser_navigate` to `http://localhost:9000{route}` — note that the console may append query params (e.g., `?page=1&perPage=50`). This is normal and expected. 2. `browser_wait_for` with `time: 3` to let the page settle (do NOT use text matching — it frequently matches hidden elements and times out) 3. `browser_snapshot` for READ-ONLY verification of page state. Do NOT use snapshot refs as click targets — refs go stale instantly on pages with live data (CPU/memory counters, pod status). Instead, use `browser_evaluate` with `document.querySelector` for all interactions: ``` browser_evaluate: () => document.querySelector('[data-test="create-button"]').click() browser_evaluate: () => document.querySelector('input[aria-label="Filter"]').value = 'test' ``` 4. `browser_take_screenshot` → `${ARTIFACTS}/<LANE>/screenshots/NN-description.png` Use **identical filenames** for both lanes. **Checking element states:** To verify a button is enabled/disabled, use `browser_snapshot` with