qa-ai-repo

Author LLM/RAG/agent evaluation suites in DeepEval that prove a feature is correct with gated numbers, not vibes. Use when asked to "eval an LLM", "test a prompt", "measure RAG quality", "check for hallucination", "score answer relevancy", "verify tool calls", or gate a release on model output quality. Ships the metric definitions baked in — faithfulness, answer relevancy, contextual precision/recall/relevancy, hallucination, tool correctness, G-Eval, bias, toxicity — with the inputs each needs, the score formula, and which direction passes. Enforces guardrails against unpinned judge models, exact-matching non-deterministic output, contaminated goldens, single-run scores, and metric-picking that ignores the real failure mode. See `reference.md` for runnable DeepEval suites and CI wiring, `tooling.md` for install/versions and alternatives.

Test web UIs for accessibility to WCAG 2.2 AA with axe-core automation plus the manual keyboard, focus, and screen-reader checks automation can't catch. Use when adding a11y assertions to a suite, auditing a page or component, wiring a11y into CI, or reviewing UI for WCAG conformance. Enforces guardrails against aria-label spam, div-soup role hacks, disabling axe rules to go green, and treating an axe pass as full coverage. See `reference.md` for setup code, rule/tag config, contrast thresholds, and a manual checklist.

Design and implement API contract tests so a provider can't break its consumers. Enforces guardrails against exact-value matching, unversioned pacts, contract tests that are secretly E2E, and deploys that ship without a can-i-deploy gate. Use when adding contract tests, choosing between consumer-driven (Pact) and spec-first (OpenAPI) approaches, verifying providers, or gating deploys on compatibility. Covers REST, GraphQL, and event/message contracts. See `reference.md` for runnable tests, broker commands, and CI wiring.

Detect, quantify, classify, and fix flaky tests — compute flake rate and per-test flake scores, reproduce non-determinism, classify the root cause, and either fix at the root or quarantine with an owner + issue + SLA. Use when tests pass-on-retry, fail intermittently in CI, or someone reaches for `retries` to make the pipeline green. Enforces guardrails against masking flake with retries, deleting tests, and ownerless quarantine. See `reference.md` for runnable rerun commands, formulas, and per-cause before/after fixes.

Design and run performance tests that prove a system meets its SLOs under realistic load — backend load/stress/soak/spike with k6 and frontend web-vitals with Lighthouse. Use when asked to "load test", "check performance", "find our capacity", "set a latency budget", or gate a release on speed and scale. Enforces guardrails against reporting averages instead of percentiles, zero-think-time hammering, extrapolating from a localhost run, trusting a single run, and ignoring error rate at load. Defines SLOs first, models a realistic workload, and turns thresholds into pass/fail CI gates. See `reference.md` for a full k6 script, stress/soak variants, Lighthouse CI budgets, and metric formulas.

Author and maintain resilient Playwright end-to-end tests using the Page Object Model, fixtures, and stable, user-facing locators. Enforces guardrails against flakiness, brittle selectors, and shared state. Use when writing, reviewing, or debugging Playwright E2E specs, or when wiring E2E tests into CI. See `reference.md` for config, fixture patterns, and commands.

Produce a tailored, risk-based QA strategy for a team or project. Use when asked to "create a QA strategy", "assess our testing approach", "build a test plan/roadmap", or decide what and how much to automate. Enforces guardrails against vanity coverage targets, big-bang rewrites, and metrics with no gate. First gathers a defined set of inputs (tech stack, team size, release cadence, current maturity, risk/compliance), then writes the strategy against a consistent template. See `reference.md` for the risk-scoring rubric, metric formulas, and quality-gate examples.

Analyze a full-stack application (frontend, backend, middleware) and design a complete test pyramid — deciding exactly which tests belong in the FE, which in the BE, which at middleware/seams, and at what level (unit, integration/component, contract, E2E). Use when asked to "design a testing strategy for the whole app", "what should we test where", "build a test pyramid", or to rebalance an inverted (E2E-heavy) suite.

Audit an existing UI/E2E test suite for overuse and recommend which tests to move down to the API (or unit) layer. Works across Playwright, WebdriverIO, and Selenium/WebDriver in any language (TS/JS, Python, Java, C#, Ruby). Enforces guardrails — relocate coverage never delete it, name a specific target endpoint/unit per demotion, keep exactly one UI happy-path per journey. Use when the suite is slow/flaky, when asked "are we testing too much through the UI", "which E2E tests should be API tests", or to fix an inverted test pyramid. See `reference.md` for before/after conversions and ready-to-run inventory commands.

Set up reliable Playwright visual regression tests with `toHaveScreenshot` — deterministic screenshots, tuned thresholds, masked dynamic regions, component-scoped snapshots, and container-generated baselines. Enforces guardrails against laptop baselines, zero-threshold noise, blind `--update-snapshots`, and whole-page goldens. Use when adding, tuning, or debugging visual/screenshot tests, or reviewing baseline diffs. See `reference.md` for config, the stabilization fixture, masking, and CI wiring.

Measure and improve the effectiveness of Jest unit tests using code coverage plus mutation testing (Stryker). Enforces guardrails against coverage theater — assertion-free tests, gamed mutation scores, ignored survivors — and gates CI on a mutation break threshold, not just coverage %. Use when asked to "check test coverage", "are our tests any good", "add mutation testing", "improve test quality", or when coverage is high but bugs still slip through. Explains why coverage alone is misleading and how to act on survived mutants. See `reference.md` for config, a worked before/after example, and commands.

Convert existing imperative Playwright tests into BDD using the playwright-bdd runner — generate Gherkin .feature files and wire step definitions to Playwright fixtures and page objects. Enforces business-language specs (declarative, not click-by-click). Use when asked to "move to BDD/Cucumber", "generate feature files from Playwright tests", or "make our E2E tests readable by the business".