pr-test-automation

name	pr-test-automation
description	Run first-pass automated manual testing for PRs that are reviewed or nearly ready to merge. Use when the user asks Codex to test a PR/branch/worktree, validate product flows, exercise browser or CLI workflows, map changed UI journeys with screenshots, verify analytics/webhooks/payments/email/SMS behavior through connected tools, or produce manual QA notes before human testing.

PR Test Automation

Overview

Validate as much of a PR as possible with local services, browser automation, CLIs, logs, and product connectors before the user does final manual testing. Treat this as a first-pass QA workflow: prove what works with evidence, identify what still needs a human, and preserve a reproducible trail.

Workflow

1. Confirm the test target:

   • Identify PR numbers, branches, worktrees, related companion PRs, and whether the user allowed rebasing/syncing.
   • Check git status, current branch, remotes, and whether unrelated local changes exist.
   • Inspect PR descriptions/review notes when they define required manual flows.

2. Check tools and authentication up front:

   • Verify required CLIs and connectors before starting long tests: gh auth status, stripe --version / active listener state, Docker status, PostHog/GitHub/Gmail connectors, cloud CLIs, or app-specific CLIs.
   • Discover verification tools before assuming a manual check is required. If Composio is available, use composio search to find candidate inbox, SMS/phone, payment, CRM, support, or provider-log tools, then inspect schemas with --get-schema before executing.
   • Prefer connected app tools for product data verification. Do not assume credentials are current.
   • Use test-mode accounts, test keys, local containers, and staging-safe endpoints unless the user explicitly asks for production verification.

3. Prepare the environment:

   • Install dependencies only where needed and report anything that changes lockfiles.
   • Start required dev servers or confirm existing sessions, ports, and mounted worktrees.
   • For companion PRs, test the combined state in the worktree/container that actually serves the code.
   • Avoid leaving duplicate background listeners or servers. List and clean up only processes started for the test.

4. Build the automated test path:

   • Use Playwright when browser behavior matters. If the repo lacks Playwright, install it in a temporary directory rather than polluting the repo.
   • Use stable, user-visible selectors first: labels, placeholders, button text, URLs, and route state.
   • Generate unique short test identities and attribution markers such as agent-e2e-<timestamp>.
   • When UI changes are in scope, map each touched surface area and user journey to screenshots in a temporary, easy-to-observe folder such as tmp/pr-<number>-qa/ or tmp/<branch>-qa/. Use ordered filenames that describe the journey step, such as 01-signup-account.png and 02-dropdown-expanded.png.
   • Capture meaningful UI states, not only final pages: empty/default, filled/selected, expanded menus, modals, validation errors, loading/success states, and at least one narrow viewport when responsive layout is likely affected.
   • Reuse the same screenshot artifact pattern for local/dev validation and, when the user asks for post-merge production verification, for production paths. Keep local and production artifacts separated by folder or filename.
   • Prefer the app's built-in test/simulation path for external effects: local inboxes, Mailhog-style UIs, fake SMS numbers, test OTP logs, sandbox payment modes, webhook listeners, or provider test keys.
   • Parse local email/SMS verification links or codes from container logs when the local environment emits them.
   • Add small human-paced waits around analytics or step-transition tests so effects and batched events have time to fire in the same order a user would experience.

5. Verify externally, not just locally:

   • Network requests prove the browser tried to send data; connector/API queries prove the product received it.
   • When simulation is unavailable, use connected recipient/provider readback: Gmail/Outlook/IMAP or email-service activity for email; Twilio/Dialpad/OpenPhone/Google Voice/test-number services or provider logs for SMS/voice; Stripe/provider dashboards for payments.
   • Query by the unique marker, test email, phone number, org ID, subscription ID, webhook event ID, request ID, or other stable test value.
   • For webhooks, confirm both CLI/listener output and backend logs, then verify downstream data.
   • For analytics dashboards, query the exact project and call out the date range and filters used.

6. Report results:

   • State what was tested, the exact test identity/marker, and the observed outcome.
   • List screenshot paths for changed UI and explain the user journey, surface area, environment, and UI state each screenshot covers.
   • When screenshots are safe to share, publish them on a repository-owned durable asset surface. For GitHub PRs, prefer an existing long-lived release such as pr-assets; do not use an arbitrary temporary host when a suitable repository release is available.
   • For GitHub PR targets where the user asked for PR testing, update the PR description with a concise QA summary where reviewers look first. Use a marked section so reruns replace the latest QA summary without overwriting the author-written description. Use a separate marked QA comment for long evidence, logs, and screenshot galleries when the description would become unwieldy.
   • Include connector/query evidence with event names, identifiers, timestamps, and important properties.
   • Separate passed automated checks from remaining manual checks.
   • Call out artifacts caused by the test harness, such as intentionally prevented navigation or mocked browser properties.

PostHog And Browser Analytics

PostHog JavaScript drops capture events from likely bots. Headless Playwright can still fetch PostHog config and run identify, while capture events are silently dropped because navigator.webdriver is true, the user agent looks automated, or navigator.userAgentData.brands includes HeadlessChrome.

When the explicit goal is to validate product analytics in local automation:

• Use a normal browser user agent.
• Mask only the automation bot signals for the test context. Setting userAgent is not enough if navigator.userAgentData still exposes headless Chrome:

const context = await browser.newContext({
  userAgent:
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
    '(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36',
})

await context.addInitScript(() => {
  const brands = [
    { brand: 'Chromium', version: '125' },
    { brand: 'Google Chrome', version: '125' },
    { brand: 'Not.A/Brand', version: '24' },
  ]
  const fullVersionList = brands.map((brand) => ({
    ...brand,
    version: `${brand.version}.0.0.0`,
  }))

  Object.defineProperty(navigator, 'webdriver', { get: () => undefined })
  Object.defineProperty(navigator, 'userAgentData', {
    get: () => ({
      brands,
      mobile: false,
      platform: 'Windows',
      getHighEntropyValues: async () => ({
        brands,
        mobile: false,
        platform: ,
        : ,
        : ,
        : ,
        : ,
        fullVersionList,
      }),
      :  ({ brands, : , :  }),
    }),
  })
})

• If the UI branches on OS or desktop/mobile, set the relevant platform signal intentionally and disclose it:

await context.addInitScript(() => {
  Object.defineProperty(navigator, 'platform', { get: () => 'Win32' })
})

• Keep the page open long enough for PostHog batching, or trigger an unload only after waiting.
• Query PostHog for the unique marker. Do not treat flags or config requests as evidence that capture events were ingested.
• If event order looks wrong under automation, rerun with human-paced waits before treating it as a product bug.

Multi-Surface Attribution Flows

Some PRs only work when a marketing site, API route, installer, desktop app, or mobile client is tested as one product flow. When attribution or install/download analytics cross those boundaries:

• Test companion PRs together in the worktree or preview environment that actually serves each surface.
• Validate route-level behavior directly before browser testing. For install/download flows, assert fresh tokens are accepted, stale or malformed tokens are dropped, invalid files or inputs are rejected, and crawler-facing routes are excluded when needed.
• Browser clipboard tests should compare visible UI text with clipboard text. It is common for the visible command/link to stay clean while the copied value includes a hidden ref, utm, or attribution token.
• If client-side analytics must create a distinct id but production capture is not part of the test, use a dummy public key and intercept analytics endpoints. If production verification is requested, use a unique marker and query the analytics project afterward.
• Use temporary app data directories for native app tests so config migrations, attribution files, cookies, and local databases do not touch the user's real profile.
• For Electron, Tauri, React Native, or similar native-shell mocks, event subscription APIs must return cleanup functions. Promise-returning mocks for on* or subscribe* APIs can create false crashes that look like product regressions.
• Direct captures that fire before an analytics SDK is fully initialized need explicit host, token, and distinct-id assertions. A request falling back to the SDK vendor default host before app config loads is usually a product bug, not enough evidence that the event will reach the intended project.
• Capture both the happy path and one negative path: accepted/refreshed attribution, stale or malformed attribution, user opt-in or opt-out, and any server-side invalid-input analytics.

External Integrations

For payment, email, SMS, analytics, and other third-party integrations:

• Confirm the account/project/mode before running tests.
• Prefer test-mode objects and fake/test cards.
• Prefer recipient/provider-side evidence over send-side success. A 200 response from the app or provider is useful but not enough when the PR's behavior depends on actual delivery, ingestion, webhook receipt, or downstream processing.
• Use plus-addresses, reserved fake phone numbers, sandbox identities, metadata, notes, UTM values, or request IDs so every external artifact can be found without ambiguity.
• Respect production stop boundaries. Do not bypass MFA, consume one-time tokens, send real calls/SMS, create paid subscriptions, charge cards, or mutate customer data unless the user explicitly approved that production action.
• Check for duplicate listeners before starting a new webhook listener.
• Record IDs that let the user or future agent find the test again: email, phone number, org key, customer ID, subscription ID, message ID, webhook event type, dashboard URL, event marker, or screenshot path.
• If a provider key lacks read scopes, try another non-destructive readback source such as a connected mailbox, recipient-side tool, provider dashboard export, app database row, webhook table, logs, or analytics event. Report the scope limitation rather than treating it as product failure.
• Never expose secrets in the final answer. Public analytics tokens are not the same as private API keys, but still describe them carefully.

Durable PR QA Descriptions, Comments, And Screenshots

When testing an open PR, preserve the result where reviewers will look first:

• Create a local artifact folder such as tmp/pr-<number>-qa/ containing raw screenshots, scripts, the exact PR Markdown, and pr-assets-manifest.json.

• Classify every image before upload. Do not upload PHI, secrets, private customer data, real inbox contents, payment details, MFA codes, production admin data, or anything inappropriate for every person who can read the PR. Keep sensitive images local and redact a copy only when the redaction can be verified visually.

• Prefer a repository-owned durable surface. For GitHub PRs, discover and reuse a published, mutable, long-lived release such as pr-assets or the repository's documented equivalent:

  repo="$(gh repo view --json nameWithOwner --jq .nameWithOwner)"
  default_branch="$(gh repo view --json defaultBranchRef --jq .defaultBranchRef.name)"
  gh release list -R "$repo" --limit 100 \
    --json tagName,name,isDraft,isPrerelease
  gh release view pr-assets -R "$repo" \
    --json tagName,isDraft,isPrerelease,isImmutable,url,assets
  

  Do not create a release per PR. Do not use an arbitrary temporary host when a suitable repository release exists.

• If no suitable release exists, creating one dedicated long-lived pr-assets release is a separate hard stop requiring an exact grant such as {"action":"create_release","repo":"owner/name","tag":"pr-assets"}. Generic GitHub, PR, comment, or asset-upload authorization does not grant creation. Use the default branch as its target and keep it out of Latest-release semantics:

  gh release create pr-assets -R "$repo" --title "PR assets" \
    --notes "Long-lived image assets for pull requests and QA evidence." \
    --latest=false --target "$default_branch"
  

  If release creation or upload is not authorized, do not fall back to a temporary host. Write the intended filenames, manifest, exact gh release create / gh release upload commands, and ready-to-paste marked PR Markdown into the artifact folder; report that durable publication is blocked.

<details open>
<summary>Signup journey screenshots</summary>

| Step | Preview |
| --- | --- |
| Account details | Shows the default account form before submission; reviewer should check required fields and spacing.<br><img src="https://example.test/01-account.png" width="420" alt="Account details form"> |
| Validation error | Shows the blocked submit state; reviewer should check copy, focus, and error placement.<br><img src="https://example.test/02-validation.png" width="420" alt="Validation error state"> |

</details>

• If PR commenting is not authorized or a connector is unavailable, write the exact Markdown comment body into the artifact folder and report the path.

Stop Conditions

Stop and ask the user before:

• Running real production payments or destructive production mutations.
• Rebasing, force-pushing, or modifying an open PR if the user has not authorized it.
• Posting comments, sending emails, toggling flags, or changing dashboards unless the user asked for that action.

Otherwise, keep going through setup, execution, verification, cleanup, and a concise result summary.

Analytics Identity Verification

Event ingestion alone is not enough — verify PERSON STITCHING whenever a PR touches analytics, signup, login, or session handling:

• Group verification queries by person_id, never by person.properties.* — event-time person properties differ per row and can make N merged users each look like "one clean person". Six merged QA users passed an email-grouped check; a person_id-grouped check exposed they were all a single person.
• Inspect raw distinct_id per event when stitching looks wrong — it names the exact identity that captured the event and usually identifies the merge vector directly.
• When the PR touches identity stitching itself (aliasing, identify calls, distinct-id or session-identity plumbing) — or the product targets shared devices — run a multi-user same-browser pass: several signups and login switches in one browser profile, then assert each user resolved to a separate person AND that functional session state (websocket auth, cookies) followed the switch. Shared-machine bugs (identity merges, stale-socket auth) are invisible to single-user passes; the pass is expensive, so reserve it for changes where that failure mode is actually in play.
• Suspect STACKED causes when a fix's re-verification still fails: fix one vector, re-run the proof, and let the raw distinct_id data name the next vector. Do not assume the fix simply "didn't work".
• Test events fired immediately before hard navigations (checkout redirects, external scheduling links): SDK batching silently drops them on unload; they need per-capture sendBeacon transport. Absence in the warehouse — not the network tab — is the proof.

Fix-Verify Loop Hygiene

• Before driving a browser proof of a just-committed fix, verify the SERVED bundle contains it (fetch the bundle URL and grep a distinctive marker, or compare the hash in page/script URLs). Dev-server rebuild races cost entire proof rounds and mimic "fix didn't work".
• Wait ~45-60s before querying an analytics warehouse for just-captured events; an empty result inside that window proves nothing.

Browser-Extension and Vendor Interference

• Password-manager extensions (1Password) steal focus into extension frames on credential-like fields; afterwards ALL automation on the tab fails with "Cannot access a chrome-extension:// URL". Prefer setting form values by element reference over click+type, dismiss popovers by clicking neutral page areas (Escape may feed the popover), and recover a wedged tab only by opening a fresh one (hosted checkout URLs resume by URL).
• Export GIF recordings BEFORE closing their tab — recordings die with the tab group.
• Vendor sandboxes rate-limit (e.g. Dropbox Sign test API throttles after ~6 signature requests/day, stalling embeds ~10 min). Budget signature-heavy passes and report throttling as an environment limit, not a product bug.

Before You Start: Head And Body Must Be Final

QA evidence is current-head evidence and the Manual tests checklist is the body's contract, so anything that would change either runs first. On a large PR (over 10 files or 300 hand-written lines) that has not had a refactor pass, say so and offer it before driving anything — a refactor landed after QA means this whole pass runs again. Likewise fresh-eyes on the PR body comes before QA, so the checklist you execute is the one the reader will see.