jaw-dev-testing

# Testing & QA Balance: ~40% Backend/API, ~40% Frontend/E2E (Playwright), ~20% Cross-cutting (CI, Security, TDD, Coverage) -- directional guidance, not a hard ratio. **Scope**: test harnesses, fixtures, mock policy, runners, Playwright, CI gates, coverage. Root-cause analysis and debugging playbooks → `dev-debugging`. - Review/test boundary and test adequacy findings: see `dev-code-reviewer`. - CI pipeline ownership and deployment verification: see `dev-devops`. - Data pipeline testing and ETL validation: see `dev-data`. - Design direction context for rendered verification: see `dev-uiux-design`. This skill activates by change surface when work needs verification depth, regression coverage, or a reproducible test harness. > **C0/C1 work (small local patches):** See `dev` §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references. > **`dev` is canonical:** `dev` §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill. ## Modular References | File | When to Read | What It Covers | |------|-------------|----------------| | `references/core/crud-test-matrix.md` | When choosing verification depth for a classified task, or testing a CRUD slice | Risk-tier minimums, per-operation negatives, UI smoke rule | | `references/edge-first-testing.md` | New unit/service/integration tests for features (skip for regression/contract tests) | Edge-first principle, test order by change type, 11-class edge matrix | | `references/backend-testing.md` | Backend/API testing | Supertest patterns, DB fixtures, auth mocking | | `references/ci-pipeline.md` | CI configuration | GitHub Actions, gates, caching, parallelism | | `references/load-testing.md` | Performance/load testing, C3+ production readiness | k6/Locust, test types, measure→profile→verify, CI gates | | `references/ml-evaluation.md` | ML model/LLM evaluation, quality gates | LLM-as-judge, RAGAS, DeepEval, CI eval gate, regression detection. CI eval gates are dataset-versioned + regression-based: pin prompts/models/retrieval config, preserve traces, calibrate judges on golden examples, fail only on meaningful regressions or safety failures | When tests depend on current external API behavior, provider docs, CI service behavior, test-environment versions, dependency audit evidence, or recorded mock/fixture sources, read the active `search` skill and follow its source-fetch and evidence-status rules. --- ## 1. Test Strategy ### 1.1 Models | Model | Best For | Emphasis | |-------|----------|----------| | Test Pyramid | monoliths, libraries | speed, isolation | | **Testing Trophy** | modern web apps, REST backends | confidence-to-cost | | Test Honeycomb | microservices, async systems | boundary verification | ### 1.2 Recommended Trophy Distribution | Layer | Default Share | Typical Tools | |-------|---------------|---------------| | Static analysis | base layer | `tsc`, ESLint, mypy, Ruff | | Unit | ~25% | Vitest, Jest, pytest | | **Integration** | **~50%** | Supertest, httpx, Testcontainers | | Contract | ~10% | Pact, OpenAPI validators, Schemathesis | | E2E | ~10% | Playwright | | Manual / exploratory | ~5% | human review | ### 1.3 Risk-First Priorities 1. auth / session / permission boundaries 2. money movement, quota, credits 3. data mutation and irreversible actions 4. file upload / parsing / external webhooks 5. shared API contracts used by frontend clients 6. error paths, retries, rollback behavior ### 1.4 Harness Selector | Problem | Primary Harness | Avoid | |---------|-----------------|-------| | pure business rule | unit / service test | browser test | | route + middleware + serialization | API integration test | mocking the route itself | | DB query / migration / transaction | real DB integration test | fake repository for SQL correctness | | frontend consuming backend JSON | contract test | manual-only verification | | rendered critical flow | Playwright smoke | asserting internal React state | | rendered artifact (visual correctness) | render-grounding loop (dev-pabcd C-RENDER-GROUNDING-01) | static parse / tsc alone | ### 1.5 General Rules - Write tests for **new features, bug fixes, refactors, and behavior changes**. - Prefer **one behavioral concern per test**. - Use factories / builders for setup; avoid repeated inline blobs. - A fast real dependency beats a mock. A mock beats an untested branch. - If the failure is mysterious, **delegate methodology to `dev-debugging`**, then return here for the regression harness. - **STRICT (TEST-ANTI-FLAKE-01):** A time-based flake is a bug. Do not use sleep-based synchronization, retry-as-fix, or green-on-retry acceptance without a deterministic cause and harness correction. - Verification depth follows `dev` §3 `DEV-VERIFY-FLOOR-01`; CRUD per-operation negative coverage is owned by `references/core/crud-test-matrix.md`. ### 1.5a Limited-Oracle / Score-Objective Evaluation When the real evaluator is scarce, paid, rate-limited, or opaque and local tests are proxy metrics for a score/objective, apply §9.5 (single owner of GATE-ORACLE-VALIDITY-01, GATE-PREFIX-HORIZON-01, GATE-INVARIANT-EV-01, GATE-HOLDOUT-LEAKAGE-01, GATE-AGREEMENT-STATS-01). Pairs with `dev-pabcd` §10 Optimization-Loop Meta-Rules. ### 1.6 Property-Based & Mutation Testing (verified 2026-07-02) | Technique | Use for | Default tools | When | |-----------|---------|---------------|------| | Property-based | Pure logic, parsers, serializers, state machines, API invariants | fast-check (TS), Hypothesis (Python) | DEFAULT for invariant-heavy code | | Mutation | Judging test-suite strength on critical domain logic, validators, security branches | Stryker (JS/TS), mutmut (Python) | Selective, after stable unit/property tests — not every PR | - Vitest 4 is the current runner baseline: Browser Mode is stable (visual regression via `toMatchScreenshot`, Playwright trace generation, `expect.schemaMatching`). --- ## 2. Backend & API Testing > Deep reference: `references/backend-testing.md` ### 2.1 Coverage Map | Layer | Verify | TypeScript Default | Python Default | |-------|--------|-------------------|----------------| | Service layer | validation, orchestration, domain errors | Vitest | pytest | | API layer | status, envelope, middleware, auth | Supertest | httpx / ASGITransport | | Repository layer | SQL / ORM correctness | Testcontainers + real DB | Testcontainers + real DB | | Background jobs | idempotency, retry, dead-letter | Vitest + fake clock | pytest + monkeypatch | ### 2.2 Mock Strategy Hierarchy ```text real deterministic dependency → Testcontainers / ephemeral infra → recorded responses / thin fake → manual stub / fake → framework mock as last resort ``` ### 2.3 Service & API Patterns Mock dependencies at service boundaries. Use Supertest/httpx for route-level integration tests. Match response envelope shape from backend contracts. ### 2.4 Database Truth with Testcontainers Use a **real database** when verifying migrations, transactions, unique constraints, foreign keys, query translation, and performance-sensitive SQL. Use Testcontainers for real DB truth in correctness-sensitive persistence tests. Start container in beforeAll/fixture setup, capture connection URI. ### 2.5 Fixture / Seed Synchronization - Prefer builders / factories over copied JSON snapshots. - Keep shared contract examples in `fixtures/contracts/` or equivalent. - Seed data should expose **stable IDs** used by Playwright smoke flows. - If frontend mocks drift from backend fixtures, write or update a **contract test first**. --- ## 3. Contract Testing Contract tests protect the **frontend↔backend boundary**. They sit between API tests and browser tests. **Rule**: Playwright proves the experience. Contract tests prove the shared shape. ### 3.1 Contract-Stable Surface - response envelope: `success`, `data`, `error`, `meta` - error taxonomy: HTTP status + machine-readable `error.code` - pagination fields, auth headers, cookie behavior - `requestId` propagation - nullability, timestamps, enums, money serialization ### 3.2 Contract Options | Style | Best For | Tooling | |-------|----------|---------| | consumer-driven contract | rapidly changing frontend/backend teams | Pact | | schema-first contract | OpenAPI-led backends | OpenAPI validators, Schemathesis | | type-level contract | TS monorepos | shared types / codegen | | full-stack smoke | final user confidence | Playwright | ### 3.3 Consumer Contract — TypeScript (Pact / PactV4) `PactV4` (aliased `Pact`) is the current interface and supports Pact Specification v4; treat `PactV3` as the legacy spec-v3 API. Workflow: 1. Define interaction: provider state + request + expected response (matchers for flexible matching) 2. Execute test against Pact mock server 3. Assert consumer expectations 4. Pact file auto-writes to `pacts/` → publish to broker → provider verifies See `references/backend-testing.md` for a full example. ### 3.4 Schema Verification Use schema-based API testing (Schemathesis) to verify OpenAPI/GraphQL contract compliance. (Dredd is legacy/inactive — do not adopt for new projects.) ### 3.5 Rules - Contract tests are **strongly recommended** for parallel FE/BE, public APIs, and cross-team contracts. - E2E success does **not** replace provider verification. - Store golden examples near the contract, not inside one app only. - If the shape is intentionally breaking, update the contract first, then all consumers. --- ## 4. Playwright Browser Testing Use Playwright after API and contract tests are already trustworthy. Browser tests should validate rendered flows, accessibility-critical interactions, and real integration seams that lower layers cannot prove alone. **Helper Scripts Available**: - `scripts/with_server.py` - Manages server lifecycle (supports multiple servers) Run scripts with `--help` first — treat as black boxes to avoid context window pollution. ### 4.1 Decision Tree: Choosing Your Approach ``` User task → Static HTML? → Read file → find selectors → write Playwright script → Dynamic app? → Server running? → No: `python scripts/with_server.py --help` → Yes: Recon-then-action (navigate → screenshot → selectors → act) ``` ### 4.2 Example: Using with_server.py ```bash # Single server: python scripts/with_server.py --server "npm run dev" --port 5173 -- python your_automation.py # Multiple servers: python scripts/with_server.py \ --server "cd backend && python server.py" --port 3000 \ --server "cd frontend && npm run dev" --port 5173 \ -- python your_automation.py ``` ### 4.3 Reconnaissance-Then-Action Pattern 1. Wait for an explicit app-ready signal or locator assertion → 2. Screenshot/inspect DOM → 3. Identify selectors → 4. Execute actions ### 4.4 Best Practices - **Use bundled scripts as black boxes** — run `--help` first, invoke directly. - Use `sync_playwright()` for synchronous scripts; always close the browser. - Prefer locator-based interactions and web-first assertions: `expect(page.get_by_role("button", name="Save")).to_be_visible()`, then `click()` on that locator. - Prefer user-facing locators, especially `get_by_role()` with an accessible name. Use `get_by_label()`, `get_by_placeholder()`, or `get_by_test_id()` when role/name cannot express the target. - Avoid `networkidle`, hard sleeps, and `wait_for_timeout()` in tests. Wait on observable app-ready signals, locator actions, or `expect()` assertions. - **AI-authored tests (DEFAULT):** Playwright MCP / Test Agents (planner/generator/healer) are generation-and-repair aids only — final acceptance still requires deterministic locators, web-first assertions, traces, and a human-readable failure artifact. ### 4.5 Reference Files - **examples/** - Examples showing common patterns: - `element_discovery.py` - Discovering buttons, links, and inputs on a page - `static_html_automation.py` - Using file:// URLs for local HTML - `console_logging.py` - Capturing console logs during automation ### 4.6 Browser Testing Rules - Run **contract tests and API tests first** for broken-data bugs. - Use Playwright for **rendered truth**, not as a replacement for service tests. - Prefer one smoke flow per critical path over many brittle micro-flows. - If a failure looks like data-shape drift, go back to **§2 Backend & API Testing** or **§3 Contract Testing**. ### 4.7 Native Computer-Use / Browse-Use QA (exploratory tier) (TEST-CU-QA-01) Browser QA loads `dev-frontend` for rendered implementation context. Playwright owns deterministic suites; native tools own immediate exploratory proof. **QA-TOOL-LADDER-01:** start at 1 and state why when skipping: 1. Manager embedded browser (`POST …/<targetId>/snapshot` for the accessibility tree, `…/screenshot`, `…/act` for click/type/scroll/key) for built or locally served web UI. 2. A full real-profile browser for login, extension, or WAF state the embedded browser cannot carry. 3. Desktop or screen-level control for GUI-only flows; keep credential entry human-supervised. 4. `agbrowse` only for public-URL response-shape proof, never built-UI driving. Use inspect -> act -> re-inspect; when DOM inspection fails, take a screenshot AND read the image back -- a screenshot produced but never read is not an observation. Evidence names the flow, states, result, and screenshots; promote durable flows to Playwright. --- ## 5. CI Pipeline Integration > Full workflow templates: `references/ci-pipeline.md` ### 5.1 Pipeline Order ```text quality (lint / typecheck) → unit + integration tests → contract tests → Playwright E2E → security scan → coverage aggregation + artifacts ``` ### 5.2 Pipeline Template Structure CI jobs in dependency chain: `quality → backend-tests → contract-tests → e2e` Key configuration: - `concurrency.cancel-in-progress: true` — avoid wasted runs - `strategy.fail-fast: false` — for matrix builds - Shard large suites: `--shard=${{ matrix.shard }}/N` - Install Playwright deps: `npx playwright install --with-deps chromium` See `references/ci-pipeline.md` for full GitHub Actions and GitLab CI templates. ### 5.3 Matrix & Parallelization | Dimension | When to Use | |-----------|-------------| | Node / Python version matrix | packages, SDKs, shared libraries | | OS matrix | native modules, CLI behavior | | shard matrix | large suites exceeding CI budget | ```bash npx vitest run --shard=1/4 npx playwright test --shard=1/4 --workers=4 pytest -n auto --dist=loadgroup ``` ### 5.4 Flaky Test Remediation | Symptom | First Fix | |---------|-----------| | passes locally, fails in CI | deterministic seeds, containerized deps, explicit waits | | order-dependent failure | reset shared state in fixtures | | green on retry only | remove wall-clock / random assumptions | | screenshot noise | stable CI image, mask dynamic regions | Protocol: detect → quarantine if blocking → assign owner → reinstate after repeated green runs. ### 5.5 Rules - Do not let Playwright be the **only** blocking job.