| name | shipwright-test |
| description | "Profile-aware test runner. Runs unit tests, smoke tests, Playwright E2E, and optional security scans.\nTRIGGER when: user wants to run tests, execute test suite, check if tests pass, run unit tests, run E2E tests, run integration tests, verify test results, check design fidelity, visual comparison, compare UI with mockup, verify against design, run visual tests, or fix failing tests.\nDO NOT TRIGGER when: user asks to write new code or implement a section (/shipwright-build), fix a bug by changing code (/shipwright-iterate), deploy (/shipwright-deploy), create requirements (/shipwright-project), plan implementation (/shipwright-plan), or design UI (/shipwright-design)." |
Shipwright Test Skill
Profile-aware test execution across all test layers.
How invoked: directly via /shipwright-test [--fix|--e2e-only|--design-fidelity|--report-boundary-coverage], or by /shipwright-run (orchestrator).
Step Index — Where the prose lives
CRITICAL: First Actions
Governing rules: Read and follow shared/constitution.md (ALWAYS / ASK FIRST / NEVER boundaries).
A. Print Intro Banner
================================================================================
SHIPWRIGHT-TEST: Test Runner
================================================================================
Runs tests across all layers based on stack profile.
Usage: /shipwright-test
or: /shipwright-test --fix (auto-fix failures, max 3 retries)
or: /shipwright-test --e2e-only (only Playwright E2E)
or: /shipwright-test --design-fidelity (only design fidelity check)
or: /shipwright-test --report-boundary-coverage (audit Affected Boundaries declarations across iterate-specs, ADR-027)
or: Invoked by /shipwright-run (orchestrator)
Test layers:
1. Unit tests (Vitest / pytest)
1.5 Integration tests (if profile has testing.integration)
1.6 pgTAP database tests (if supabase/tests/database/ exists)
2. Smoke test (HTTP 200 on DEV URL)
3. Playwright E2E (if UI project + DEV URL available)
3.6 Cross-page consistency (if .shipwright/designs/visual-guidelines.md exists)
3.7 Design fidelity (if .shipwright/designs/screen-routes.json exists)
3.8 Performance budget (if profile has testing.performance.enabled)
3.9 Stop dev server (always, finally-block — runs even if 3.7 / 3.8 failed)
4. Security scan → out-of-band; see /shipwright-security (not auto-invoked)
================================================================================
B. Detect Profile
Read shipwright_project_config.json from project root and load profile from
{plugin_root}/../../shared/profiles/{profile}.json. If no config, detect from
package.json / pyproject.toml. Full detection rules: see
first-actions.
B2. Detect Invocation Mode
Resolve it with {shared_root}/scripts/tools/get_phase_context.py --phase-task-id "{phaseTaskId}" --phase test (omit --phase-task-id entirely if the orchestrator did not hand you one — that is what selects standalone) and store the returned mode as invocation_mode.
The dispatch token the orchestrator hands you is the authority — never re-derive
the mode from run-config state. pipeline → full integration. standalone (no
token) → add "mode": "standalone" to shipwright_test_results.json, skip pipeline
state updates, and ASK first when requires_out_of_sequence_warning is true.
error → STOP (never continue as standalone: the validator rejects
standalone-stamped results and the run deadlocks).
Full rules: see first-actions.
B3. Load Project Context
Read .shipwright/agent_docs/architecture.md and previous
shipwright_test_results.json. Missing files: skip silently.
B4. Prerequisite Self-Healing
Auto-generate missing dev_url, .shipwright/designs/visual-guidelines.md,
.shipwright/designs/screen-routes.json, .shipwright/planning/claude-plan-e2e.md,
and playwright.config.ts where source data exists. Full rules and per-artifact
commit hints: see prerequisite-self-healing.
C. Determine Test Strategy
Based on profile:
| Profile | Unit Runner | E2E | Smoke URL Pattern |
|---|
supabase-nextjs | npx vitest run | Playwright | http://localhost:3000 |
| (future) | configurable | configurable | configurable |
Step 0: Phase Session Context Recovery
If the orchestrator handed you a phaseTaskId, run
get_phase_context.py --phase-task-id <id> as the first action.
Full body: see step-0-phase-session.
Step 1: Run Unit Tests
uv run "{plugin_root}/scripts/lib/test_runner.py" --profile "{profile}" --layer unit
Autonomous mode auto-applies --fix behavior. Max 3 retries, structured
debugging (root cause -> pattern check -> hypothesis -> fix -> re-run).
Full body: see step-1-unit-tests.
Step 1.5: Run Integration Tests
Skip if no testing.integration profile config OR tests/integration/ missing.
CI: missing env vars = FAIL; locally: skip with warning. Fast-fail on
ECONNREFUSED/ETIMEDOUT and on >50% simultaneous failure. Never auto-fix by
weakening RLS, swapping to service-role, or disabling URL safety.
Full body: see step-1.5-integration-tests.
Step 1.6: Run pgTAP Database Tests
Skip if supabase/tests/database/ missing. Run via supabase test db or the
shared test_runner.py --layer pgtap. Autofix as for integration.
Full body: see step-1.6-pgtap-tests.
Step 2: Run Smoke Test (if DEV URL available)
uv run "{shared_root}/scripts/smoke_test.py" --url "{dev_url}" --timeout 10 --health-path "/api/health"
DEV URL sources (in order): shipwright_build_config.json.dev_url ->
SHIPWRIGHT_DEV_URL env -> http://localhost:3000 default.
The smoke-test script is shared ({shared_root}/scripts/smoke_test.py), not a
project file. On failure: diagnose, autonomous fix, retry, then ASK after 2.
Full body: see step-2-smoke-test.
Step 2.5: Generate E2E Specs from Plan + check coverage per journey
Coverage is checked per journey, always — journey_coverage.py reports each
planned journey as covered or uncovered, so a journey added to the plan after
the first spec exists no longer goes unreported. Greenfield gaps block;
brownfield gaps file a follow-up routed to /shipwright-adopt. Generation is
still skipped when e2e/ already has .spec.ts files (and entirely when the
profile has no UI); otherwise scan .shipwright/planning/ for
claude-plan-e2e.md and generate e2e/flows/*.spec.ts, e2e/pages/*.page.ts,
e2e/fixtures/*, and e2e/fixtures/auth.setup.ts. Full body and structure: see
step-2.5-e2e-spec-generation.
Step 3: Run Playwright E2E (if applicable)
Prereqs: UI profile + smoke test passed. Run playwright_setup.py,
dev_server.py start, then playwright_runner.py --cwd {project_root}.
Auto-fix groups failures by root cause (auth / selector / data / network /
other), fixes per group with commits, parks unresolvable groups after 3
attempts and ASKs the user.
Full body: see step-3-playwright-e2e.
Step 3.5: E2E Results Verification
Read e2e-results.json (Playwright authoritative). Filter to the chromium
project (exclude setup). Reconcile shipwright_test_results.json.e2e with
Playwright's expected + unexpected + flaky + skipped, carry flaky +
flaky_tests into the record, write an e2e_verification_note if divergent,
and verify playwright-report/index.html. Counting is per test, not per
attempt: a test that only passed on retry is a pass, counted separately.
Full body: see step-3.5-e2e-verification.
Step 3.6: Cross-Page UI Consistency Check (if applicable)
Runs if .shipwright/designs/visual-guidelines.md exists AND profile has UI
(component_library set), or standalone via --consistency. Non-blocking
(WARNING). Outliers are grouped by Spacing / Components / Colors root cause and
fixed with per-group commits, max 3 retries per category.
Full body: see step-3.6-ui-consistency.
Step 3.7: Design Fidelity Verification — Regressions-Only (if applicable)
Runs if .shipwright/designs/screen-routes.json exists, or standalone via
--design-fidelity. Non-blocking (WARNING). Triages needs_review screens
against design-fidelity-report.json into Resolved / Regression / Persistent
Failure / Unchecked. Deep-reviews flagged screens against 5 dimensions, max
3 retries each.
Full body: see step-3.7-design-fidelity.
Step 3.8: Performance Budget Check (if applicable)
Runs if profile.testing.performance.enabled: true OR
shipwright_test_config.json.performance.enabled: true. Lighthouse via
Playwright Chromium; bundle measurement = gzip-compressed *.js/*.css in
build_output_dir. Gate warn (default) vs block (opt-in).
Override precedence: --gate -> shipwright_test_config.json.performance ->
profile -> defaults (LH 85, bundle 250 KB gz, LCP 2500 ms, gate warn).
Full body: see step-3.8-performance-budget.
Step 3.9: Stop Dev Server (always — finally-block semantics)
Run unconditionally as a cleanup pass — even if Step 3.7 or Step 3.8
raised or failed. A blocked test phase is recoverable; a zombie dev server
bound to the dev port is not.
uv run "{plugin_root}/../../shared/scripts/dev_server.py" stop --cwd {project_root}
Full body: see step-3.9-stop-dev-server.
Step 3.95: Boundary Coverage Report (if --report-boundary-coverage)
Runs only when invoked with --report-boundary-coverage OR
shipwright_test_config.json.boundary_coverage.enabled: true. Audit hook for
ADR-024. Scans .shipwright/planning/iterate/**/*.md for ## Affected Boundaries, correlates against shipwright_events.jsonl, writes coverage
summary + drift signals. Non-blocking (WARNINGs in test summary).
Full body: see step-3.95-boundary-coverage.
Step 4: Security Scan → /shipwright-security
Security scanning runs out-of-band; /shipwright-run does NOT auto-invoke
it (decoupled in iterate sec-report-and-orchestrator-decouple, 2026-04).
This step is a no-op in shipwright-test.
Full body: see step-4-security.
Step 5: Report Results
First run warning_followups.py over the finished record so every non-blocking
layer leaves a tracked follow-up (Step 5.0) — browser tests, consistency and
fidelity now do what the performance budget always did. Then print the
SHIPWRIGHT-TEST RESULTS summary (unit / integration / pgTAP / smoke / e2e /
journeys / consistency / design fidelity / performance / security / overall) and
the preview banner when the profile has UI. Failures declared in
shipwright_known_failures.json — the same list the audit phase reads — are
reported as known-and-accepted, separately from genuine failures. Capture learnings into
conventions.md#Learnings. Record test_run + phase_completed events,
update build dashboard, write canon-marker handoff, append phase_history,
then mark the test step complete via the orchestrator.
Per-layer enforcement: see results-enforcement.
Completion gate (every layer must have an explicit result): see
completion-gate.
Full body: see step-5-report-results.
Anti-Rationalization
See anti-rationalization — the table of
common rationalizations ("Tests pass = correct", "E2E flaky = ignore",
"We'll add more later", etc.) and the reality check for each.
