shipwright-test

name

shipwright-test

description

"Profile-aware test runner. Runs unit tests, smoke tests, Playwright E2E, and optional security scans.\nTRIGGER when: user wants to run tests, execute test suite, check if tests pass, run unit tests, run E2E tests, run integration tests, verify test results, check design fidelity, visual comparison, compare UI with mockup, verify against design, run visual tests, or fix failing tests.\nDO NOT TRIGGER when: user asks to write new code or implement a section (/shipwright-build), fix a bug by changing code (/shipwright-iterate), deploy (/shipwright-deploy), create requirements (/shipwright-project), plan implementation (/shipwright-plan), or design UI (/shipwright-design)."

Shipwright Test Skill

Profile-aware test execution across all test layers.

How invoked: directly via /shipwright-test [--fix|--e2e-only|--design-fidelity|--report-boundary-coverage], or by /shipwright-run (orchestrator).

Step Index — Where the prose lives

Section	Reference
First Actions (A. Banner / B. Detect Profile / B2. Mode / B3. Context / B4. Prereqs / C. Strategy)	first-actions · prerequisite-self-healing
Step 0: Phase Session Context Recovery	step-0-phase-session
Step 1: Run Unit Tests	step-1-unit-tests
Step 1.5: Run Integration Tests	step-1.5-integration-tests
Step 1.6: Run pgTAP Database Tests	step-1.6-pgtap-tests
Step 2: Run Smoke Test	step-2-smoke-test
Step 2.5: Generate E2E Specs from Plan + per-journey coverage	step-2.5-e2e-spec-generation
Step 3: Run Playwright E2E	step-3-playwright-e2e
Step 3.5: E2E Results Verification	step-3.5-e2e-verification
Step 3.6: Cross-Page UI Consistency Check	step-3.6-ui-consistency
Step 3.7: Design Fidelity Verification	step-3.7-design-fidelity
Step 3.8: Performance Budget Check	step-3.8-performance-budget
Step 3.9: Stop Dev Server (always — finally-block)	step-3.9-stop-dev-server
Step 3.95: Boundary Coverage Report	step-3.95-boundary-coverage
Step 4: Security Scan -> /shipwright-security	step-4-security
Step 5: Report Results	step-5-report-results · results-enforcement · completion-gate
Anti-Rationalization	anti-rationalization
Test layers strategy (legacy)	test-layers
Reflection	reflection

---

CRITICAL: First Actions

Governing rules: Read and follow shared/constitution.md (ALWAYS / ASK FIRST / NEVER boundaries).

A. Print Intro Banner

================================================================================
SHIPWRIGHT-TEST: Test Runner
================================================================================
Runs tests across all layers based on stack profile.

Usage: /shipwright-test
   or: /shipwright-test --fix        (auto-fix failures, max 3 retries)
   or: /shipwright-test --e2e-only   (only Playwright E2E)
   or: /shipwright-test --design-fidelity  (only design fidelity check)
   or: /shipwright-test --report-boundary-coverage  (audit Affected Boundaries declarations across iterate-specs, ADR-027)
   or: Invoked by /shipwright-run (orchestrator)

Test layers:
  1.  Unit tests (Vitest / pytest)
  1.5 Integration tests (if profile has testing.integration)
  1.6 pgTAP database tests (if supabase/tests/database/ exists)
  2.  Smoke test (HTTP 200 on DEV URL)
  3.  Playwright E2E (if UI project + DEV URL available)
  3.6 Cross-page consistency (if .shipwright/designs/visual-guidelines.md exists)
  3.7 Design fidelity (if .shipwright/designs/screen-routes.json exists)
  3.8 Performance budget (if profile has testing.performance.enabled)
  3.9 Stop dev server (always, finally-block — runs even if 3.7 / 3.8 failed)
  4.  Security scan → out-of-band; see /shipwright-security (not auto-invoked)
================================================================================

B. Detect Profile

Read shipwright_project_config.json from project root and load profile from {plugin_root}/../../shared/profiles/{profile}.json. If no config, detect from package.json / pyproject.toml. Full detection rules: see first-actions.

B2. Detect Invocation Mode

Resolve it with {shared_root}/scripts/tools/get_phase_context.py --phase-task-id "{phaseTaskId}" --phase test (omit --phase-task-id entirely if the orchestrator did not hand you one — that is what selects standalone) and store the returned mode as invocation_mode. The dispatch token the orchestrator hands you is the authority — never re-derive the mode from run-config state. pipeline → full integration. standalone (no token) → add "mode": "standalone" to shipwright_test_results.json, skip pipeline state updates, and ASK first when requires_out_of_sequence_warning is true. error → STOP (never continue as standalone: the validator rejects standalone-stamped results and the run deadlocks). Full rules: see first-actions.

B3. Load Project Context

Read .shipwright/agent_docs/architecture.md and previous shipwright_test_results.json. Missing files: skip silently.

B4. Prerequisite Self-Healing

Auto-generate missing dev_url, .shipwright/designs/visual-guidelines.md, .shipwright/designs/screen-routes.json, .shipwright/planning/claude-plan-e2e.md, and playwright.config.ts where source data exists. Full rules and per-artifact commit hints: see prerequisite-self-healing.

C. Determine Test Strategy

Based on profile:

Profile	Unit Runner	E2E	Smoke URL Pattern
supabase-nextjs	npx vitest run	Playwright	http://localhost:3000
(future)	configurable	configurable	configurable

---

Step 0: Phase Session Context Recovery

If the orchestrator handed you a phaseTaskId, run get_phase_context.py --phase-task-id <id> as the first action. Full body: see step-0-phase-session.

Step 1: Run Unit Tests

uv run "{plugin_root}/scripts/lib/test_runner.py" --profile "{profile}" --layer unit

Autonomous mode auto-applies --fix behavior. Max 3 retries, structured debugging (root cause -> pattern check -> hypothesis -> fix -> re-run). Full body: see step-1-unit-tests.

Step 1.5: Run Integration Tests

Skip if no testing.integration profile config OR tests/integration/ missing. CI: missing env vars = FAIL; locally: skip with warning. Fast-fail on ECONNREFUSED/ETIMEDOUT and on >50% simultaneous failure. Never auto-fix by weakening RLS, swapping to service-role, or disabling URL safety. Full body: see step-1.5-integration-tests.

Step 1.6: Run pgTAP Database Tests

Skip if supabase/tests/database/ missing. Run via supabase test db or the shared test_runner.py --layer pgtap. Autofix as for integration. Full body: see step-1.6-pgtap-tests.

Step 2: Run Smoke Test (if DEV URL available)

uv run "{shared_root}/scripts/smoke_test.py" --url "{dev_url}" --timeout 10 --health-path "/api/health"

DEV URL sources (in order): shipwright_build_config.json.dev_url -> SHIPWRIGHT_DEV_URL env -> http://localhost:3000 default. The smoke-test script is shared ({shared_root}/scripts/smoke_test.py), not a project file. On failure: diagnose, autonomous fix, retry, then ASK after 2. Full body: see step-2-smoke-test.

Step 2.5: Generate E2E Specs from Plan + check coverage per journey

Coverage is checked per journey, always — journey_coverage.py reports each planned journey as covered or uncovered, so a journey added to the plan after the first spec exists no longer goes unreported. Greenfield gaps block; brownfield gaps file a follow-up routed to /shipwright-adopt. Generation is still skipped when e2e/ already has .spec.ts files (and entirely when the profile has no UI); otherwise scan .shipwright/planning/ for claude-plan-e2e.md and generate e2e/flows/*.spec.ts, e2e/pages/*.page.ts, e2e/fixtures/*, and e2e/fixtures/auth.setup.ts. Full body and structure: see step-2.5-e2e-spec-generation.

Step 3: Run Playwright E2E (if applicable)

Prereqs: UI profile + smoke test passed. Run playwright_setup.py, dev_server.py start, then playwright_runner.py --cwd {project_root}. Auto-fix groups failures by root cause (auth / selector / data / network / other), fixes per group with commits, parks unresolvable groups after 3 attempts and ASKs the user. Full body: see step-3-playwright-e2e.

Step 3.5: E2E Results Verification

Read e2e-results.json (Playwright authoritative). Filter to the chromium project (exclude setup). Reconcile shipwright_test_results.json.e2e with Playwright's expected + unexpected + flaky + skipped, carry flaky + flaky_tests into the record, write an e2e_verification_note if divergent, and verify playwright-report/index.html. Counting is per test, not per attempt: a test that only passed on retry is a pass, counted separately. Full body: see step-3.5-e2e-verification.

Step 3.6: Cross-Page UI Consistency Check (if applicable)

Runs if .shipwright/designs/visual-guidelines.md exists AND profile has UI (component_library set), or standalone via --consistency. Non-blocking (WARNING). Outliers are grouped by Spacing / Components / Colors root cause and fixed with per-group commits, max 3 retries per category. Full body: see step-3.6-ui-consistency.

Step 3.7: Design Fidelity Verification — Regressions-Only (if applicable)

Runs if .shipwright/designs/screen-routes.json exists, or standalone via --design-fidelity. Non-blocking (WARNING). Triages needs_review screens against design-fidelity-report.json into Resolved / Regression / Persistent Failure / Unchecked. Deep-reviews flagged screens against 5 dimensions, max 3 retries each. Full body: see step-3.7-design-fidelity.

Step 3.8: Performance Budget Check (if applicable)

Runs if profile.testing.performance.enabled: true OR shipwright_test_config.json.performance.enabled: true. Lighthouse via Playwright Chromium; bundle measurement = gzip-compressed *.js/*.css in build_output_dir. Gate warn (default) vs block (opt-in). Override precedence: --gate -> shipwright_test_config.json.performance -> profile -> defaults (LH 85, bundle 250 KB gz, LCP 2500 ms, gate warn). Full body: see step-3.8-performance-budget.

Step 3.9: Stop Dev Server (always — finally-block semantics)

Run unconditionally as a cleanup pass — even if Step 3.7 or Step 3.8 raised or failed. A blocked test phase is recoverable; a zombie dev server bound to the dev port is not.

uv run "{plugin_root}/../../shared/scripts/dev_server.py" stop --cwd {project_root}

Full body: see step-3.9-stop-dev-server.

Step 3.95: Boundary Coverage Report (if --report-boundary-coverage)

Runs only when invoked with --report-boundary-coverage OR shipwright_test_config.json.boundary_coverage.enabled: true. Audit hook for ADR-024. Scans .shipwright/planning/iterate/**/*.md for ## Affected Boundaries, correlates against shipwright_events.jsonl, writes coverage summary + drift signals. Non-blocking (WARNINGs in test summary). Full body: see step-3.95-boundary-coverage.

Step 4: Security Scan → /shipwright-security

Security scanning runs out-of-band; /shipwright-run does NOT auto-invoke it (decoupled in iterate sec-report-and-orchestrator-decouple, 2026-04). This step is a no-op in shipwright-test. Full body: see step-4-security.

Step 5: Report Results

First run warning_followups.py over the finished record so every non-blocking layer leaves a tracked follow-up (Step 5.0) — browser tests, consistency and fidelity now do what the performance budget always did. Then print the SHIPWRIGHT-TEST RESULTS summary (unit / integration / pgTAP / smoke / e2e / journeys / consistency / design fidelity / performance / security / overall) and the preview banner when the profile has UI. Failures declared in shipwright_known_failures.json — the same list the audit phase reads — are reported as known-and-accepted, separately from genuine failures. Capture learnings into conventions.md#Learnings. Record test_run + phase_completed events, update build dashboard, write canon-marker handoff, append phase_history, then mark the test step complete via the orchestrator.

Per-layer enforcement: see results-enforcement. Completion gate (every layer must have an explicit result): see completion-gate. Full body: see step-5-report-results.

---

Anti-Rationalization

See anti-rationalization — the table of common rationalizations ("Tests pass = correct", "E2E flaky = ignore", "We'll add more later", etc.) and the reality check for each.