accessibility-test-harness

name

accessibility-test-harness

description

Design the accessibility verification harness — WCAG-oriented coverage of keyboard operability (tab order, visible focus, no traps), accessible names/labels/roles, focus management in dialogs and route changes, color contrast, and screen-reader expectations (announcements, landmarks, live regions) — as BOTH automated tooling (axe-style scans in component and E2E layers, contrast checks in CI) AND a manual checklist for what automation cannot judge. Findings map to WCAG criteria with severity and evidence. Use when asked to test/verify accessibility, set up a11y checks or CI scanning, write a keyboard-only or screen-reader test pass, or when an a11y complaint/audit obligation lands. Produces harness design, configs, checklists; execution handed to the wired layers. Do NOT use for a general UI walkthrough (clickthrough-test-engineer), general manual cases (manual-test-case-creator), implementing a11y FIXES, or non-a11y evidence policy (screenshot-evidence-planner).

Accessibility Test Harness

Purpose

Produce the harness that verifies accessibility instead of asserting it: automated scans placed at the right layers (component tests, E2E pass, CI) for what machines can catch, plus a manual keyboard/screen-reader checklist for what they cannot — with every check tied to a WCAG criterion, a severity rubric, and evidence conventions. Automation finds roughly a third of real a11y failures; the harness is honest about which third and covers the rest manually.

Use When

• Use when: asked to test, verify, or "check" accessibility of an app, feature, or component set.
• Use when: setting up automated a11y scanning (axe-style) in component tests, E2E, or CI.
• Use when: a keyboard-only or screen-reader verification pass is requested, or an audit/complaint/legal obligation (WCAG/EN 301 549/ADA-adjacent) needs a repeatable verification harness.
• Use when: a design system or form framework needs per-component a11y acceptance checks.
• Do NOT use when: the ask is a general UI walkthrough for breakage — clickthrough-test-engineer (this harness is the a11y-specialized pass).
• Do NOT use when: writing general manual test cases — manual-test-case-creator (a11y checklist items follow ITS stranger-executable format but live here).
• Do NOT use when: implementing the FIXES (component/markup changes) — that is product work reviewed by code-reviewer; this skill verifies.
• Do NOT use when: general screenshot policy — screenshot-evidence-planner (a11y evidence follows its rules).

Inputs to Inspect

1. The target standard and obligation: WCAG version/level (2.1/2.2, AA typical) from compliance requirements, contracts, or default to AA — stated explicitly.
2. The surfaces in scope: routes, critical journeys, the component library (design-system components multiply coverage — one accessible Button fixes a thousand instances).
3. Existing automation layers to wire into: component test setup (vitest-unit-component-engineer conventions), E2E suite (playwright-e2e-engineer), CI tiers (qa-automation-architect blueprint).
4. Known a11y state: prior audit findings, existing lint rules (eslint-plugin-jsx-a11y-style), testid-fallback flags from the Playwright suite (each is a missing-semantics lead).
5. Assistive-tech expectations: which screen reader × browser combos the product commits to (defaults in the reference).

Workflow

1. Pin the standard and scope. WCAG version + level + the surface list (critical journeys first, component library in parallel). Every later finding cites its criterion.
2. Split checks: automated vs manual — explicitly. Machines catch missing names/labels, role misuse, contrast (mostly), duplicate ids, ARIA validity; humans judge focus order sensibility, announcement quality, keyboard flow usability, context changes. The split table lives in references/a11y-checklists.md; every check lands on exactly one side (or both, with reason).
3. Design the automated tier: axe-style scan assertions in component tests (fail on new violations per component), an E2E a11y pass scanning each critical-journey page state, contrast checking against the token palette in CI, and a11y lint at the PR tier. Baseline-then-ratchet for existing violations: freeze the count, forbid new ones, burn down.
4. Design the manual tier as executable checklists: keyboard-only pass (reach everything, visible focus, no traps, logical order, ESC/arrow conventions), screen-reader smoke per committed combo (landmarks, form labels/errors announced, dialog focus containment + announcement, live regions for async results), zoom/reflow at 200/400%, contrast spot checks automation can't see (text over images, states). Each item: steps, expected, WCAG ref — stranger-executable per manual-test-case-creator format.
5. Define severity + evidence: blocker (journey impossible by keyboard/ SR) → cosmetic; evidence per finding (what, where, criterion, how reproduced, capture per evidence rules).
6. Place in CI and cadence: component scans on PR (blocking on NEW violations), journey scan on merge/nightly, manual passes per release and on a11y-relevant changes; wire ownership.
7. Hand off execution: scan wiring lands via the engineer skills (manual-only where they run browsers/builds); manual passes to testers / clickthrough-test-engineer sessions; fixes to product owners — this skill re-verifies after.

Output Format

A11Y HARNESS — <scope>
Standard: <WCAG version/level + obligation source>
Scope: <journeys, routes, components>
Automated tier:
  <layer (component/E2E/CI-lint/contrast) → tool/config → violation policy
   (baseline + ratchet) → CI placement>
Manual tier:
  <checklist id> — <keyboard|screen-reader|zoom|contrast-judgment> —
  steps/expected/WCAG ref — cadence
AT matrix: <screen reader × browser combos committed>
Split rationale: <what automation covers vs cannot — explicit>
Severity rubric: <blocker→cosmetic with a11y-specific definitions>
Evidence: <per evidence-planner rules; finding format with WCAG citation>
Ownership & cadence: <who runs what, when; re-verification loop>
Handoffs: <wiring → engineer skills; fixes → product/code-reviewer>

Validation Checklist

• WCAG version/level pinned and every check cites a criterion.
• Automated/manual split is explicit; nothing automation can't judge is claimed as automated coverage.
• Keyboard checklist covers reach-everything, visible focus, no traps, order, and dialog/ESC behavior.
• Screen-reader checks name the committed AT × browser combos.
• New-violation policy (baseline + ratchet) defined for automated scans.
• Severity rubric is a11y-specific (keyboard-impossible ≠ cosmetic).
• Findings format includes criterion, reproduction, and evidence.
• No component fixes implemented here.

Gotchas

• Automated scans passing ≠ accessible: axe-style tools catch ~30-40% of WCAG failures; claiming "accessible, scans green" is the classic false-confidence failure — the manual tier is not optional.
• ARIA is a repair tool, not a feature: scan-driven ARIA sprinkling (role/aria-label on everything) often makes screen-reader UX worse; prefer native semantics findings.
• Focus "visible" per contrast math can still be practically invisible — human judgment item, not just a token check.
• Contrast tools miss text over images/gradients and state colors (hover/disabled) — keep them in the manual judgment list.
• jsdom-based scans can't evaluate visibility/contrast reliably — put those checks at the browser layer, not the component layer.
• Testid fallbacks in the E2E suite are missing-semantics leads: a control automation can't target by role, AT users can't target either.

Stop Conditions

• No standard/level can be pinned (no obligation, no default accepted) → ask; findings without a criterion baseline are opinions.
• The committed AT matrix is unknown and screen-reader checks would be speculative → propose the default matrix and get it confirmed.
• Scan wiring requires running builds/browsers now → hand to the manual-only engineer skills rather than executing here.
• The scope is a full formal conformance audit (VPAT/legal) → that is a specialist engagement; this harness feeds it but doesn't replace it — say so.

Supporting Files

• references/a11y-checklists.md — the automated-vs-manual split table, keyboard and screen-reader checklists with WCAG refs, AT matrix defaults, severity rubric.
• evals/evals.json — trigger + behavior cases.
• evals/trigger-evals.json — discrimination within the UI/manual cluster (a11y vs general walkthrough vs general cases).