test-roadmap

name

test-roadmap

description

Phase 4 of the test-readiness pipeline. Synthesizes the three prior artifacts (inventory, system design, migration) into a single executable roadmap `test-readiness-plan.md` with five sequenced milestones and an agent-executable task list. Includes mandatory sections on where we are now, where we want to be, the gap, the speed delta, and open risks. Output is suitable for handoff to an engineer or to `shipwright /dev-task`. Invoke when the `/test-roadmap` command runs.

test-roadmap skill

Purpose

Synthesize. Take the three phase artifacts and produce the single document that drives execution.

When invoked

By the /test-roadmap command. Requires:

• docs/test-readiness/test-inventory.md
• docs/test-readiness/test-system.md
• docs/test-readiness/test-migration.md

Output structure

docs/test-readiness/test-readiness-plan.md has six sections:

1. Where we are now

Distilled from Phase 3:

• Layer coverage map (counts: how many tests at each layer, how many ought to be there)
• Local-runnable status: % of tests that run locally with no network
• Canary status: size of current canary suite vs. target (from Phase 2 critical-path roster)
• Speed status: current p95 per layer vs. budget
• "Rebuild" debt: count by layer and effort
• "Delete (redundant)" count and what it implies about false-confidence coverage

2. Where we want to be

Distilled from Phase 2:

• Framework matrix per layer
• Local-substitute map
• Coverage targets per tier
• Canary suite definition (critical-path roster)
• Speed budgets per layer

3. The gap

The concrete diff between sections 1 and 2:

• Missing layers (e.g., "no integration tests at all")
• Wrong-layer tests (count and severity)
• External-only deps with no local substitute (blockers)
• Canary suite size vs. target
• Speed delta — current per-layer p95 vs. target, expressed as ratio or time-to-close

4. Roadmap — five milestones

Always five, in this order. Each milestone has a clear definition-of-done (DOD) that gates the next.

Milestone 1: Infrastructure baseline

Set up runners, local substitutes, CI pipeline shape. No new tests written yet — just the rails.

• DOD: bun test (or stack equivalent) runs at every layer with the Phase 2 framework. Docker-compose / testcontainers boot all named deps. CI runs the unit + integration + smoke layers under budget.

Mandatory M1 task — naming convention + runner-exclusion config: Milestone 1 must always include an explicit task to establish the test file naming convention and runner-exclusion configuration. File-discovery collisions (the canary config picking up local tests, or Jest running Playwright specs) cause cascading failures across the roadmap. Establish the rules before writing any tests.

This task covers:

1. File-naming convention — define the suffix rules for each entry point:
   • .canary.ts / .canary.spec.ts — canary-only tests (Playwright project or filtered script)
   • .smoke.ts / .smoke.spec.ts — smoke layer (pre-merge local run)
   • .integration.ts / .integration.spec.ts — integration layer (testcontainers / docker-compose)
   • .unit.ts / .unit.spec.ts or default (no suffix) — unit layer Use one consistent convention across the repo; document it in a docs/test-readiness/naming.md or equivalent.
2. Runner-exclusion config — each runner must explicitly exclude the other layers' files:
   • Jest / Vitest testMatch / include — exclude .canary., .smoke., .integration. files from the unit runner
   • Playwright testMatch — scope to .canary.spec.ts or the canary project directory only; exclude all .unit., .integration. files
   • Coverage config exclude — exclude test helpers, fixtures, and non-source files from coverage reports
   • bun test --filter — use path-based filters that respect the naming convention
3. Verification — run each entry point in isolation and confirm it picks up only its layer's files (e.g., bun test --filter integration finds zero unit files).

Mandatory M1 task — carried-forward measurement-only items: if Phase 3's test-migration.md flags a measurement-only item (not a test file) as carried forward unactioned across 3 or more consecutive cycles, Milestone 1 must include it as the first task, by construction — do not rely on noticing the streak in prose when assembling the task list.

Milestone 2: Critical-path coverage

Write or rebuild every critical tier test across all layers.

• DOD: 100% of inventory items tagged critical have a passing test at the prescribed layer. delete (redundant) tests in this tier are gone.

Milestone 3: Canary suite live

Smoke + E2E canary-eligible tests run green against a freshly deployed environment. Gated on deploy_model (determined in Process step 1): this milestone only applies when deploy_model == 'staged'.

• DOD (when deploy_model == 'staged'): CI job runs the canary suite against a staging URL post-deploy. Suite completes in <60s. All canary tests pass.
• When deploy_model != 'staged' (i.e. declared direct, declared none, or undeclared): there is no staging/canary pipeline to gate on. Replace this milestone's entire content with a single note line — canary not applicable -- deploy_model={value}, where {value} is direct, none, or undeclared — and skip the milestone entirely: no DOD, no tasks. Do not fabricate canary-plumbing work for a pipeline that doesn't exist.

Milestone 4: High-tier coverage

Fill high tier gaps.

• DOD: 100% of high tier inventory items have coverage at the prescribed layer.

Milestone 5: Cleanup

Delete or refactor remaining rebuild and delete (redundant) tests; remove false-confidence coverage. This milestone comes LAST because deletions are safer once new coverage is in place.

• DOD: zero tests in rebuild bucket remain. Zero tests in delete (redundant) bucket remain. CI passes.

5. Task list

Flat, ordered. Each task has:

• ID (sequential, e.g., T-001)
• Milestone (1–5)
• Files to touch (paths)
• Layer (unit / integration / smoke / E2E / infra)
• Bucket origin (reuse / promote / rebuild / delete / net-new)
• Expected outcome (one line)
• Verification command (the exact CLI invocation that proves it's done)

This format is designed to be consumed by shipwright /dev-task as a queue.

Task sizing for agent execution: a task is agent-executable as written iff all three of:

1. Independently deployable — the PR for this task can merge without the next task. No "part 1 of 2" that leaves the system in a broken intermediate state.
2. Single concern — one functional unit per task. A task touching auth middleware AND a user factory AND two smoke tests is three tasks. Single concern ≠ one operation repeated across N services. Applying the same change to N services is N tasks — each service has its own files, its own verification command, and its own PR surface area.
3. Hard cap: ~1000 lines changed. Larger PRs slow review, increase merge conflict risk, and reduce the agent's confidence in its own output. If a task exceeds this, break it at a natural seam — usually by layer (infra task, then unit task, then integration task) or by feature area.

N-service refactors are N flat tasks, wired with real dependencies (non-negotiable): Building on rule 2 above — one operation repeated across N services is N tasks, not one — each of those N tasks is emitted as its own independently-titled row in the task list (section 5), with its own files, its own verification command, and its own PR surface area. There is no parent/summary task and no synthetic "closes when its children close" row: task-store has no structural parent/child field, only dependencies, and encoding "parent closes when children close" as free text has broken in production. Wire the N tasks together with real dependencies entries only where genuine technical ordering exists between them (e.g. one task's output is literally imported by another) — never merely because they touch "the same" logical refactor. If no such ordering exists, the tasks are fully independent and each carries an empty dependencies array; do not invent a dependency just to show the field is used.

Example — 6-service auth middleware refactor:

Instead of one oversized task:

T-042 | M2 | services/*/src/middleware/auth.ts (6 files) | infra | rebuild
      | Migrate auth middleware to new token format
      | bun test --filter auth

Emit 6 flat, independently-titled tasks — one per service, no parent/P-NNN row:

T-042a | M2 | services/payments/src/middleware/auth.ts      | infra | rebuild
       | Migrate auth middleware — payments (introduces shared validateToken() helper)
       | bun test --filter payments/auth

T-042b | M2 | services/users/src/middleware/auth.ts         | infra | rebuild
       | Migrate auth middleware — users (imports validateToken() from payments)
       | bun test --filter users/auth
       | depends_on: T-042a

T-042c | M2 | services/notifications/src/middleware/auth.ts | infra | rebuild
       | Migrate auth middleware — notifications
       | bun test --filter notifications/auth

T-042d | M2 | services/billing/src/middleware/auth.ts       | infra | rebuild
       | Migrate auth middleware — billing
       | bun test --filter billing/auth

T-042e | M2 | services/catalog/src/middleware/auth.ts       | infra | rebuild
       | Migrate auth middleware — catalog
       | bun test --filter catalog/auth

T-042f | M2 | services/search/src/middleware/auth.ts        | infra | rebuild
       | Migrate auth middleware — search
       | bun test --filter search/auth

Here, payments is the first service migrated and introduces a shared validateToken() helper that users' middleware genuinely imports — so T-042b's row carries a depends_on: T-042a annotation (test-fix's Step 2.3 parses this and Step 5.4 computes the task-store dependencies array from it). The remaining four services (notifications, billing, catalog, search) have no real ordering dependency on any other service's migration, so their rows carry no depends_on: annotation and they end up with an empty dependencies array, running in parallel. This is not advisory — do not emit the oversized task and place it in Open risks. Apply the split during task generation. Open risks is for genuinely ambiguous human calls, not for tasks the sizing algorithm knows are too large.

6. Open risks

Anything the audit couldn't determine without a human call. Common entries:

• Tests in the delete (redundant) bucket where the canonical owner is itself untested (don't delete until owner exists)
• External dep with no clean local substitute (does the team accept a recorded-fixture-only approach?)
• Tests at the wrong layer that may be doing important work the inventory didn't capture (human verification needed)

Process

1. Read all three prior artifacts. Abort if any missing. Also determine deploy_model using the same read-with-fallback mechanism skills/test-inventory/SKILL.md Step 1 uses ("CLAUDE.md deploy-model declaration"): check the target repo's root CLAUDE.md for a ## Deploy model section with value staged, direct, or none; if the section is absent, treat it as undeclared and default to NOT staged. Store this value — Step 5 gates Milestone 3 canary-task generation on it.

   Using the same read mechanism, also determine recorded_fixture_automation: check the target repo's root CLAUDE.md for a ## Recorded-fixture automation section with value planned or not planned; if the section is absent, treat it as undeclared and default to planned (preserves existing behavior for repos that haven't opted out). Store this value — Step 5 gates recorded-fixture-refresh task generation on it.

2. Extract metrics: layer counts, speed numbers, bucket counts.

3. Compute the gap (section 3 math).

4. Determine the starting T-NNN offset (task-store query — run before any ID is assigned). T-NNN numbering must continue across cycles, not restart at T-001 every time /test-roadmap runs — a restart collides with test-fix's task-store IDs from a prior cycle (test-fix maps T-NNN → test-t-{nnn}-{repo-slug}), which is how a second cycle's roadmap has duplicated a first cycle's already-shipped work under fresh numbering.

   Derive repo (org/repo, e.g. app-vitals/shipwright) from git remote get-url origin — strip the https://github.com/ (or [e-mail]:) prefix and the .git suffix. Derive repo-slug as the last path segment, lowercased (e.g. shipwright).

   Query task-store for every existing task in this repo, regardless of status — a task from a prior cycle that has since merged still reserved its ID and must not be reused. Pass an explicit high limit (task-store defaults to 50 per page, which silently truncates the scan on a repo with a longer test-readiness history):

   curl -sf -H "Authorization: Bearer $SHIPWRIGHT_TASK_STORE_TOKEN" \
     "$SHIPWRIGHT_TASK_STORE_URL/tasks?repo={url-encoded-repo}&limit=500" | jq '.tasks'
   

   If the response's .total exceeds the number of tasks returned, repeat with &offset={n*500} until every task has been scanned.

   From the combined results, filter to task id values matching test-t-<NNN>-{repo-slug} (the ID shape test-fix writes — see skills/test-fix/SKILL.md Step 5). Extract the numeric <NNN> from each match and take the max: max_existing. Set starting_offset = max_existing + 1 — or 1 (i.e. T-001) if no matching tasks exist, which is the correct behavior for a repo's first test-readiness cycle.

5. Generate the task list by walking the migration buckets in this order, assigning sequential T-NNN IDs starting from starting_offset (step 4) rather than always restarting at T-001:

   • Milestone 1: all infra items + paired repo-config tasks (see pairing rule below). Exception: when recorded_fixture_automation == 'not planned' (Process step 1), omit every recorded-fixture-refresh-loop item — the .github/workflows/recorded-fixture-refresh.yml workflow task, its paired GitHub-Environment/secrets task, and any per-service "record fixtures for <service>" / "wire recorded-fixture-refresh cron for <service>" tasks from repo-config/SKILL.md's deferred pattern — regardless of what test-migration's bucket table flags as a gap. Replace them with a single note in the roadmap's Milestone 1 section: recorded-fixture automation not applicable -- recorded_fixture_automation=not planned.
   • Milestone 2: all critical tier items (net-new + rebuild + promote)
   • Milestone 3: only when deploy_model == 'staged' (Process step 1) — all canary-eligible items needing canary plumbing. When deploy_model != 'staged' (declared direct, declared none, or undeclared), emit zero Milestone 3 tasks here — the section 4 note (canary not applicable -- deploy_model={value}) is the only artifact of this milestone; do not generate canary-plumbing tasks for a pipeline that doesn't exist.
   • Milestone 4: all high tier items (net-new + rebuild + promote)
   • Milestone 5: all delete (redundant) items + remaining rebuild cleanup + plugin feedback collector

6. Apply the pairing rule from ${CLAUDE_PLUGIN_ROOT}/skills/repo-config/SKILL.md: every task that creates or modifies a CI workflow file MUST emit a paired branch-protection task that depends_on the workflow task. Without this, the audit ships as advisory rather than enforced. The pairing rule is non-negotiable; skipping it is the failure mode the user will catch and the plugin will be blamed for.

7. Apply the E2E classification guardrail (non-negotiable): Before emitting any task with layer: e2e, verify against test-system.md's "Classifying a new test" step that the proposed test journey actually exercises a real browser (step 4: "Does it test a multi-step browser flow? → e2e (Playwright)"). If the journey is a backend orchestration flow, an API contract test, or any other non-browser interaction, downgrade the task to layer: integration or layer: smoke with a one-line note explaining why (e.g., "backend orchestration flow — moved to smoke"). This guardrail prevents shipping e2e tasks that violate the test classification rules, ensuring the e2e layer contains only true multi-step browser-driven flows.

8. Load ${CLAUDE_PLUGIN_ROOT}/assets/templates/test-readiness-plan.md.tmpl. Fill. Write to docs/test-readiness/test-readiness-plan.md.

   • {{M3_HEADING_AND_DOD}} is gated on deploy_model (Process step 1) — the template has no static text for this milestone precisely because it must reflect the gate:
     • When deploy_model == 'staged': fill with ### Milestone 3 — Canary suite live\n**DOD:** Smoke + E2E canary-eligible tests run green against the deployed env in <60s wall time.
     • When deploy_model != 'staged' (declared direct, declared none, or undeclared): fill with the single note line — ### Milestone 3 — Canary suite live\n**DOD:** canary not applicable -- deploy_model={value} (no task list, no DOD beyond the note) — where {value} is direct, none, or undeclared.

Failure modes to avoid

• Don't sequence Milestone 5 (cleanup) before Milestone 2 (critical-path). Deleting a "redundant" test before its canonical owner exists creates a coverage hole. Always build before deleting.
• Don't skip the speed delta. It's the most actionable single number for "are we converging."
• Don't write a roadmap that's a copy of the migration table. The roadmap is sequenced and milestone-gated; the migration is unsorted bucketing. The synthesis is the value.
• The plugin feedback collector task must not create a per-repo log inside the plugin. plugins/shipwright/ is repo-agnostic and ships publicly (see root CLAUDE.md) — a file that names the target repo and accumulates dated, repo-specific entries (e.g. references/pipeline-learnings.md) bakes private/internal repo names into the OSS package as soon as a second repo runs the pipeline. Route real feedback through learning-capture's existing mechanism instead: a generalized, repo-agnostic instruction edited directly into the relevant skill/doc, or — for autonomous agents, which don't use learning-capture interactively — a # Harness TODO entry for the dream job to fold in per its normal routing. Never a new narrative file under the plugin's own references/.
• Don't restart T-NNN numbering at T-001 on a repo with a prior cycle. Always run the step 4 task-store query first. Skipping it silently collides with test-fix's test-t-{nnn}-{repo-slug} IDs from an earlier cycle and produces a roadmap that duplicates already-shipped work under fresh numbering.
• Don't generate Milestone 3 canary-plumbing tasks when deploy_model != 'staged'. Check the value determined in Process step 1 before emitting any Milestone 3 task. Getting this wrong is not hypothetical: on 2026-07-14 this happened to shipwright's own repo (deploy_model: direct — no staging environment) and produced a pile of canary-plumbing tasks for a pipeline that doesn't exist, requiring 8 tasks to be manually cancelled. When not staged, emit the canary not applicable -- deploy_model={value} note in section 4 and skip the milestone — zero tasks, not a smaller pile of tasks.
• Don't bundle "build a never-proven capability" and "automate it on a schedule" into one task. This is not hypothetical either: one target repo's cycle-3 roadmap asked for the whole recorded-fixture-refresh loop (~15 services, ~8 net-new production credentials) in a single task, which sat pending/HITL for multiple cycles with no path forward — the repo had never actually exercised a live RECORD=1-style capture against a real third-party API; fixtures were all hand-authored. Wiring real credentials into CI to automate a process that has never once been proven manually is exactly the failure mode repo-config/SKILL.md's "deferred — not implemented" note is meant to block. Check recorded_fixture_automation (Process step 1) before emitting any recorded-fixture-refresh task; when it's not planned, emit zero tasks per the Milestone 1 exception above rather than one monolithic task.