cd-test-architecture

# CD Test Architecture Role: worker. This command assesses and reports — it does not write tests or refactor code; it hands the migration steps to `/plan` or `/build`; Step 4b's build-vs-document ask is this skill's one interactive branch point, and even there it proposes a Story rather than invoking `/build` itself. ## Overview An **advisory, application-level** skill: it assesses how an existing application is tested, classifies that against a CD-aligned test taxonomy, finds the tests that can't run in a clean CI gate, and recommends a target architecture plus a migration path. It does not write tests or refactor code. Where `test-design-advisor` works at the unit/module level and `test-smell-review` finds smells in test files, this skill works at the **whole-application** level: test types, pipeline stages, and per-component patterns. Grounded in these knowledge references — read the first two before assessing: - `knowledge/cd-test-architecture.md` — the six test types, the determinism→pre-merge-gate rule, the adapter rule, double validation, pipeline stages, and MinimumCD-vs-Fowler terminology. - `knowledge/component-test-patterns.md` — per-component patterns (UI / Services / Batch) with isolation strategy and pipeline placement. - `knowledge/database-test-patterns.md` — load when a component is database-backed: Database Sandbox isolation, Transaction Rollback / Table Truncation Teardown, and the rule that pushes most data-logic tests onto in-memory Fakes so they stay pre-merge-gate eligible. - `knowledge/test-doubles.md` — the Stub/Fake/Mock/Spy taxonomy; load when Step 4b's database or downstream-service branch proposes a hand-rolled Fake double, so it's named correctly (a Fake, never a "mock"). - `knowledge/virtual-service-libraries.md` — the per-stack record-and-replay virtual-service library catalog, its preferred-vs-backup framing, and its Resolution order (existing-tool detected → catalog default → operator override/decline); load when Step 4b's Downstream-service branch resolves a `Build (Fake)` row's library choice. ## Constraints - Advisory only. Assess and recommend; do not edit production or test code. Hand the migration steps to `/plan` or `/build` — including Step 4b's build-vs-document ask: it proposes a downstream Story, it never invokes `/build` or edits code directly. - Use MinimumCD vocabulary (unit / component / contract / integration / E2E / static analysis) consistently; when the codebase uses other names, map them explicitly. - The pre-merge gate may contain **only deterministic** tests (static, unit, component, contract). Any test that needs a database, broker, downstream service, or environment secrets configured to run is, by definition, not a pre-merge test — flag it. - Recommend isolation via in-memory doubles (or, for downstream-service adapters, a record-and-replay virtual-service library per `virtual-service-libraries.md`) + owned adapters, validated by the double-validation loop. Do **not** recommend standing up the whole system (docker-compose of dependencies) for the gate — that is the configured-dependency problem this architecture removes. - **Do not assume provider cooperation.** For dependencies the team doesn't control, the defense against contract breakage is consumer-owned, scheduled verification against the provider's test environment (out-of-band) plus consumer resilience — not provider-side CDC verification. Recommend accordingly. - **Baseline before refactor (don't lead with refactoring).** For under-tested or legacy components, first recommend the best outside-in test achievable *at existing seams without changing the code* — a characterization baseline — then the refactor that improves testability under that green baseline. Never change behavior and structure in the same step. Defer the procedure to the `legacy-code` skill and use the DDD skills (`domain-driven-design`, `domain-analysis`) to suggest the target structure for the refactor. - Minimal tooling: prefer a record-and-replay virtual-service library (`virtual-service-libraries.md`) or in-memory doubles for the pre-merge gate, one real browser for UI, testcontainers only for off-gate adapter integration. Don't recommend a sprawl of frameworks. - Be concise: tables and ordered steps, not prose. Cite the knowledge file instead of restating it. ## Parse Arguments Arguments: a target application/repo path or description. Optional `--component <name>` to scope to one component, `--ci <path>` to point at the existing pipeline config, **`--external-tests <path-or-repo-or-description>`** to point at tests that live outside this repo (another repo's suite, a third-party runner, Postman/Insomnia collections, manual test scripts, recorded UI flows, spreadsheets of test cases), and `--stack <id>` to override stack detection (default: detect from manifests; resolved profile key like `dotnet`, `node`, `spring-boot`, `go`, `django`, `react`, `vue`, `ssr-htmx`). `--yes` runs unattended: Step 4b's build-vs-document prompt is skipped and every flagged component defaults to Document-only, per `human-oversight-protocol`'s non-interactive convention. If little or no in-repo testing is found and no external location is given, **ask** where the application is actually tested before concluding it is untested. If no target is given, ask for one. ## Steps ### 0. Detect stack Resolve the project's stack so the assessment can resolve concrete tools from `knowledge/test-stack-profiles/<stack>.md`. Mirrors `skills/test-design-advisor/SKILL.md:31, 62`: when `--stack <id>` is provided, **it takes precedence** and detection is skipped. Otherwise, read manifests at the target — `package.json` (refined to react/vue via dependency, or to ssr-htmx when an htmx dep is present alongside `templates/*.html`), `*.csproj` / `*.sln`, `pom.xml` / `build.gradle*`, `go.mod`, `pyproject.toml` / `requirements.txt` — and resolve the matching profile key. Load `knowledge/test-stack-profiles/<stack>.md` (and any references it points at) so the *Target architecture* table can name concrete tools per layer. When no profile matches, proceed with stack-agnostic guidance and **name the missing profile in the report** — never block on it. ### 1. Inventory the application's components Map the deployable/testable surfaces and assign each its pattern from `component-test-patterns.md` (User Interface; API Provider / API Consumer / Event Consumer / Event Producer / Stateful Service / CLI-Library; Scheduled Job). A real system is usually several — list each surface. **Exhaustive surface-type discovery is the mandatory default — always on, never operator-gated (issue #1464).** Every invocation of this step actively searches for each pattern type in `component-test-patterns.md`'s table (Step 1's own list above already names all eight — not a separate list to invent here), not just the first or most obvious one found. Stopping after the first obvious surface type — e.g. finding a couple of HTTP routes and concluding the inventory is complete — is a spec violation of this step, not an acceptable shallow default: a real system frequently has surfaces in more than one pattern group at once (Services *and* Batch *and* UI, say), and each group must be actively searched for, not merely recorded when stumbled upon. **Record the search itself, not only what it found**: in the Output's `### Components & patterns` table (Step 6), include a row for every pattern type actively searched for — even one with no matching surface (state `None found` rather than omitting the row) — mirroring `/gherkin-derive` Step 2's `## Analysis Coverage` section (issue #1450), so a thin run is detectable from the report rather than merely asserted here. **Existing-tool detection (feeds Step 4b's Downstream-service branch).** While inventorying, also check the target's dependency manifest and test files for an existing virtual-service/record-replay library (e.g. Nock, WireMock, WireMock.Net, VCR.py, go-vcr). When found, note it: per `knowledge/virtual-service-libraries.md`'s Resolution order, it becomes the default recommended tool for that component's `Build (Fake)` row in Step 4b instead of the catalog's per-stack default, with no switch suggested. This is inventory-phase data gathering, not a Step 4 recommendation. **Graph-assisted inventory.** Prefer CodeGraph/Repowise over raw `Grep` for locating components, their deployable surfaces, and existing test suites — see [`knowledge/codegraph-vs-graphify.md`](../../knowledge/codegraph-vs-graphify.md) for tool selection and the fallback contract. ### 2. Inventory the existing tests and classify them Find the test suites **in the repo** and classify each against the six types in `cd-test-architecture.md`. For each, record: type, what it actually exercises, whether it is deterministic, and **what it requires to run** (DB URL, broker, downstream service, secrets, sleep, real clock). If in-repo tests are sparse or absent, the application is not necessarily untested — it may be tested out-of-repo (see Step 2b — locate and harvest out-of-repo tests). Do not conclude "no tests" without checking. ### 2b. Locate and harvest out-of-repo tests When `--external-tests` is given (or in-repo tests are sparse and the user points you to external coverage), treat the external location as the **current specification of intended behavior** and harvest it: - **Other-repo automated suites** — read the suites; classify them by type just like in-repo tests; note that they live outside the component's repo and pipeline. - **Postman / Insomnia / `.http` collections** — each request + its assertions describes an API contract and a scenario. Extract: endpoint, request shape, expected response, and which success/failure scenario it covers. - **Manual test scripts / spreadsheets / recorded UI flows** — extract each step as a behavior the team cares about (a candidate component/E2E scenario), and note it is currently human-executed and non-repeatable. Produce a behavior inventory from these sources, mapped to the component patterns from Step 1 (inventory the application's components). This becomes the **basis for improvement** — the behaviors to re-express as deterministic, in-repo, gated tests. ### 3. Diagnose CD-fitness gaps Flag, with evidence: - **Out-of-repo / third-party-runner testing** — the component's tests live in another repo, a separate QA runner, Postman collections, or manual scripts rather than alongside the code. **This is an anti-pattern**, even when that external coverage is extensive: the tests cannot gate the component's own merges, are not versioned with the code they verify, are usually non-deterministic and environment-coupled, and silently drift from the code. The goal state is deterministic tests co-located with the code and run in its pipeline. Flag this explicitly and treat the external suite as the *source material* (Step 2b — locate and harvest out-of-repo tests), not the destination. - **Manual / non-repeatable testing** — behavior verified only by humans following scripts. Non-repeatable, unsuitable for any gate; each such script is a behavior to automate. - **Mis-typed gate tests** — "unit/component" tests that require a real dependency or are non-deterministic (real clock/RNG/network/sleep). These cannot be a pre-merge gate. - **Configured-dependency tests** — tests that need the rest of the system stood up to run. - **Coverage gaps** — behavior (success + failure modes per the component pattern) not covered at any deterministic layer. - **Drift risk** — doubles with no validation loop. In particular, assume **no provider cooperation**: a contract that nobody runs against the real provider on a schedule is undefended. Flag the absence of *consumer-owned, scheduled provider-contract verification in a test environment* — relying on provider-side CDC verification is not sufficient for providers you don't control. - **No resilience to a broken contract** — the consumer assumes the provider holds; no tests that it survives a provider break (timeout, retry/backoff, circuit breaker, drifted/malformed response). Assume the provider *will* break without versioning. - **Inverted shape** — reliance on integration/E2E where component/contract tests would gate deterministically. ### 3b. Find testable seams and the achievable outside-in baseline For each under-tested or untested component, identify the **testable seams** — places where behavior can be observed or substituted without editing the code (HTTP handler, CLI entrypoint, message handler, exported function, existing injection points; object seams via interfaces/polymorphism, link seams via DI/module substitution). Then state the **best outside-in test achievable right now without refactoring** — a characterization test at the outermost reachable seam that locks in current behavior. This is the immediate, zero-risk baseline, distinct from the (later) clean CD gate. See `cd-test-architecture.md` → Outside-In First, and the `legacy-code` skill. ### 4. Recommend the target architecture Per component, using its pattern: which test types cover which layers, **what to double to run pre-merge without configuration**, the success scenarios and failure modes to cover, the double-validation loop, and the pipeline stage for each (pre-merge gate vs Stage 1/2 vs out-of-band vs post-deploy). Show the resulting pre-merge gate is deterministic and config-free. For under-tested components, separate the recommendation into **(a) the outside-in characterization baseline writable today without refactoring** and **(b) the post-baseline refactor** toward this target (use the DDD skills to suggest where boundaries/seams should land). **E2E justification gate** — canonical in `knowledge/cd-test-architecture.md#the-e2e-justification-gate`. For each E2E recommendation, document that all four conditions hold, and record one sentence per behavior explaining why E2E was *not* chosen when the first three cover it. **The pyramid is a cost heuristic, not a target shape** — canonical in `knowledge/cd-test-architecture.md#the-pyramid-is-a-cost-heuristic-not-a-target-shape`. Per-component / per-behavior placement is the valid output; if the suite shape is pathological, name the pathology and the behaviors that suffer from it — never propose a numeric redistribution. ### 4b. Build-vs-document decision (off-gate adapter test doubles) When Step 4 identifies one or more components needing an off-gate adapter test double (today: a testcontainers-based real-DB test, or a downstream-service adapter test), ask the operator once per run, batched across every such component regardless of adapter kind — this is the shared prompt every adapter-kind slice appends to, not a new prompt each run; the Downstream-service branch below, including its record-and-replay-library sub-question, is one such append, already shipped, not future work. The one prompt lists every such component and asks the operator, for each one, to choose exactly one of three options — there is no separate follow-up sub-question (a `Build (Fake)` row's construction-method sub-question is carried in this same prompt and reply — see the Downstream-service branch below): 1. **Build (testcontainers)** — propose a downstream Story for a real-dependency test using testcontainers (exact Story shape per the branch subsection below). 2. **Build (Fake)** — propose a downstream Story for a hand-rolled Fake double (exact Story shape per the branch subsection below). 3. **Document-only** (default) — the recommendation lands in the report as today, with no further action. The operator answers with one of these three choices for each listed component in a single reply — one prompt is surfaced per run, carrying a per-component three-way answer, not one verdict applied to every component in the batch. Non-interactive runs (per `human-oversight-protocol`'s `--yes` / `DEV_TEAM_AUTO_APPROVE=1` / no-TTY convention) skip the prompt entirely — no prompt is surfaced, and every such component's recommendation lands in the report only, document-only, exactly as today. An ambiguous or absent answer for any component in the batched three-way question — for example "maybe", "I'm not sure", a bare "yes" or "no", an unqualified "build" that does not name which of the two Build options (none of these map to one of the three labeled options), or silence/empty input — defaults to Document-only for that component, same as any other ambiguous answer: never guessed, never blocked on. Repos with no such gap see no behavior change: no prompt is asked, and the report is unchanged. #### Database-specific branch For the database-IS-the-SUT band, the three-way answer directly determines the outcome per component: - **Build (testcontainers)** — propose a Story titled `[<component>] Add testcontainers-based real-DB test`. Its description names Database Sandbox isolation and both teardown options — Transaction Rollback and Table Truncation — and cites `database-test-patterns.md`. - **Build (Fake)** — propose a Story titled `[<component>] Add hand-rolled Fake database double`. Its description names the Fake as an in-memory repository implementing the same interface, per `test-doubles.md`'s Fake taxonomy, cites both `database-test-patterns.md` and `test-doubles.md`, and always carries this caveat verbatim: "Caveat: this hand-rolled Fake cannot verify actual SQL, mapping, or schema correctness the way a real-engine test can — a deliberate coverage trade-off, not a silent downgrade." - **Document-only** — the recommendation lands in the report only, as today; no further action. #### Downstream-service branch Applies to API Consumer, Event Consumer, and Event Producer components (`component-test-patterns.md`) that Step 4 flagged for the same off-gate adapter-test-double decision as the database branch above — added to the *same* batched per-run prompt, not a second prompt: the shared prompt above already states later adapter-kind slices append here, and this is that append. Every row offers all three options identically, with no ownership-based distinction: `Build (testcontainers)`, `Build (Fake)`, `Document-only`. **All off-gate validation of a downstream-service test double runs out-of-band, on a schedule — never pre-merge/in-band — regardless of whether the dependency is team-controlled or third-party.** This applies uniformly to the testcontainers-based adapter-integration test, the hand-rolled Fake's companion provider-contract verification, and the library-backed double's re-record/diff companion action alike, per `component-test-patterns.md`'s and `cd-test-architecture.md`'s Integration-test placement rule: a real dependency is non-deterministic regardless of who owns it, so it never earns an in-band exception. An ambiguous or absent answer for a downstream-service row defaults to `Document-only`, the same ambiguous-answer rule as any other ambiguous answer above — not a new rule. **When the top-level answer for a row is `Build (Fake)`, a second, narrower sub-question resolves which construction method to use — asked in the same single batched prompt and reply as the top-level three-way choice above, never a follow-up round-trip.** The sub-question applies only to rows where the operator chose (or defaulted to) `Build (Fake)`; a `Build (testcontainers)` or `Document-only` row carries no such sub-question, and any install/decline preference volunteered for either of those rows is ignored, not an error. The answer grammar is explicit and three-valued, not binary: (a) accept the recommended virtual-service library (the existing-tool-detected or catalog default, per `virtual-service-libraries.md`'s Resolution order), (b) name a specific different library, or (c) decline (hand-rolled Fake, the backup). **An ambiguous or absent sub-answer defaults to (a), installing the recommended library — the preferred solution — not (c) decline.** This is a distinct, narrower default rule from the top-level three-way choice's own ambiguous-answer default stated above, which remains `Document-only`, unchanged: the library is preferred and hand-rolled is backup, so the safer default when this narrower sub-question is unspecified is the preferred option, not the fallback. An operator's explicit-accept answer (e.g. "Build (Fake), accept") installs the recommended library identically to this ambiguous/absent-answer case. For an Event Consumer/Producer row whose broker protocol has no protocol-appropriate virtual-service library (per `virtual-service-libraries.md`'s broker-tooling note), this sub-question is not presented at all — the row states plainly that no such library exists for its protocol, and defaults to the hand-rolled Fake Story below, with no tool name substituted. Each selected option proposes a Story, following the database branch's shape: - **Build (testcontainers)** — propose a Story titled `[<component>] Add testcontainers-based adapter integration test`. Its description exercises the real outbound client against a testcontainers-hosted, protocol-compatible instance, states the test runs out-of-band on a schedule (never pre-merge/in-band), and cites the relevant `component-test-patterns.md` pattern (API Consumer / Event Consumer / Event Producer — name whichever applies). (Maintainer note, not emitted Story content: #1435 deliberately scoped the record-and-replay library-preference sub-question below to the `Build (Fake)` row only — this bullet carries no tool-selection language, enforced by `test_sub_question_only_applies_to_build_fake_row` in `tests/skills/test_cd_test_architecture_virtual_service_libraries.py`.) - **Build (Fake)** — propose a Story titled `[<component>] Add hand-rolled Fake downstream-service double`. Its description names a hand-rolled Fake behind a team-owned thin adapter, per `test-doubles.md`'s Fake taxonomy, and cites `component-test-patterns.md`. When the sub-answer above resolves to (c) decline, the Story always carries this caveat verbatim: "Caveat: this hand-rolled Fake cannot verify that the adapter actually satisfies the real service's wire contract — pair it with scheduled provider-contract verification against the provider's real environment, per the API Consumer / Event Consumer / Event Producer patterns." The Story always — no ownership-based exemption — additionally proposes scheduled, out-of-band provider-contract verification against the dependency's real environment as a companion action, not a fourth option. When the sub-answer above resolves to installing a library instead (default, named override, or explicit accept), the Story title instead becomes `[<component>] Add <tool-name>-backed downstream-service double` (e.g. `[<component>] Add Nock-backed downstream-service double`), naming the concrete tool chosen, and its description states the library-backed double runs in-band (pre-merge); this library-backed outcome carries its own one-line caveat in place of the hand-rolled-Fake wire-contract caveat above — no wire-contract caveat is needed for a library-backed double, because the library's own off-gate re-record/diff validation (below) is what keeps it honest, not a hand-rolled assumption about the wire contract; the same scheduled provider-contract-verification companion action described above is unchanged in count — still one, not a second, additional off-gate test artifact — only its *mechanism* changes: for a library-backed double, that action is the library's own re-record/diff capability against the real dependency (when the chosen tool has one), in place of a hand-rolled contract test. When the chosen library has no re-record/diff capability, the companion action reverts to the hand-rolled provider-contract verification described earlier in this bullet — still one artifact, not a second one. - **Document-only** — the recommendation lands in the report only, as today; no further action. ### 5. Produce a migration path Order the moves from current → target, lowest-risk first. The spine is **baseline before refactor**: get behavior under test at existing seams *without changing code*, then refactor under that green baseline (never behavior + structure in one step). When tests are out-of-repo (Step 2b — locate and harvest out-of-repo tests), the harvested behaviors feed that baseline. Typical full sequence: 1. **Characterization baseline (no refactoring)** — at the outermost reachable seam, write outside-in tests that lock in current behavior; harvest any out-of-repo/manual behaviors into this inventory and reproduce them here. Get green. 2. **Introduce owned adapters and seams — under the baseline** (Adapter Rule; `testability-patterns.md`; DDD skills suggest where boundaries/seams belong). Refactor only with the baseline green. 3. Add in-memory doubles + deterministic **component tests** reproducing the baselined behaviors. 4. Add **contract tests** pinning request/response boundaries. 5. Add **consumer resilience tests** (survive a provider break). 6. Add **scheduled provider-contract verification** against a test environment.