runtime-validation

# Runtime Validation This skill serves two phases of the pipeline: - **Design phase** (Part 1) — planning roles read this to produce a testing strategy before implementation begins - **Validation phase** (Part 2) — tester reads this to execute startup verification + write and run tests --- ## Part 1: Testing Strategy > **Audience:** planning roles producing the pre-implementation plan. Tester reads the resulting testing strategy during validation and executes it. > **Output:** a "Testing Strategy" section in the planning artifact, produced before writing test code. ### 1.1 Test Tiers Not every project needs every tier. Pick the tiers that match the app's architecture: | Tier | Config | DB | Security | Speed | When to use | |------|--------|----|----------|-------|-------------| | Unit | Mocked deps | None | None | 1ms | Always — implementer writes & runs. Test infrastructure is designed here (see §1.4 item 10) | | Slice | Partial app context | In-memory | Mocked | 100ms | Framework-supported isolation (e.g., @WebMvcTest, partial DI) | | Integration | Full app context + Docker-based infra | Real DB | Real | 3s | **Default for most apps** — catches 80% of bugs | | E2E (API) | Running app (random port or external) | Real | Real | 5s | API-only apps, or API layer of mixed apps | | E2E (Browser) | Playwright against running app | Real | Real | 10s | **Any app with rendered HTML** — only tool that sees what users see | **Key insight:** Integration tier (full app context + Docker-based real infrastructure) is the sweet spot for most migrations. Don't skip it by jumping straight to browser E2E. ### 1.2 Tool Selection | App Type | Primary Tool | Reference | |---|---|---| | Server-rendered Java (JSP, Thymeleaf) with minimal JS | Playwright (if Node available) + framework integration tests + Docker-based infra | [playwright.md](references/playwright.md) | | REST API only (no browser UI) | Framework integration tests + Docker-based infra | [rest-assured.md](references/rest-assured.md) | | Client-side SPA (React, Angular, Vue) | Playwright (TypeScript) | [playwright.md](references/playwright.md) | | Server-rendered with significant client-side JS | Playwright + framework integration tests for API layer | Both | | Messaging (Kafka / Service Bus) | Docker-based infra (Testcontainers for Java, docker-compose for others) + framework test runner | [testcontainers.md](references/testcontainers.md) | | Mixed (API + SPA browser UI) | Playwright for UI + framework integration tests for API | Both | | .NET (ASP.NET Core) | `WebApplicationFactory<T>` + HttpClient | (built-in) | | .NET (WinForms/WPF — no web UI) | xUnit + in-process testing | (no browser E2E needed) | **Non-Java frameworks — equivalent tools:** | Framework | Integration Tool (like MockMvc) | Browser E2E | Startup Check | |-----------|-------------------------------|-------------|---------------| | .NET (ASP.NET Core) | `WebApplicationFactory<T>` + `TestServer` | Playwright | `dotnet run` + health check | | Node.js (Express/Nest) | supertest (calls app instance directly) | Playwright | `npm start` + health check | | Python (Django) | `django.test.Client` | Playwright | `manage.py runserver` | | Python (FastAPI) | `TestClient` (httpx-based) | Playwright | `uvicorn` + health check | | Go | `httptest.NewServer` + `http.Client` | Playwright | compile + run + health check | | Ruby (Rails) | `ActionDispatch::IntegrationTest` | Playwright | `rails server` | The pattern is universal: each framework has a built-in "test the handler without a real HTTP server" tool for the integration tier, and Playwright covers the browser tier across all of them. **When is Playwright REQUIRED vs RECOMMENDED?** - **REQUIRED**: SPAs with client-side routing/state (React, Angular, Vue) — no other tool exercises the full client-side stack - **STRONGLY RECOMMENDED**: Server-rendered HTML apps (JSP, Thymeleaf, Razor) — Playwright catches view rendering bugs invisible to MockMvc/TestRestTemplate - **FALLBACK (no Node.js)**: `@SpringBootTest` + `@AutoConfigureMockMvc` covers 80% of flows (cross-controller, forward/redirect, security). Gap: cannot verify actual view rendering. Document this gap but do not skip testing. **Spring MVC apps with HTML views: use MockMvc, not TestRestTemplate.** `@SpringBootTest` + `@AutoConfigureMockMvc` loads all controllers + security + exception handlers and can assert view names, model attributes, and forwarded URLs. `TestRestTemplate` cannot inspect these — a `@ControllerAdvice` that catches exceptions and returns 200 + error view will make TestRestTemplate see all requests as "successful". **For apps with HTML views, always prefer MockMvc.** **Spring REST API apps (JSON only): TestRestTemplate is fine.** API responses use HTTP status codes directly (200/4xx/5xx), no view resolution to inspect. TestRestTemplate handles cookies/sessions/redirects naturally and is simpler to use. ### 1.3 Environment Decision Tree > **⚠️ MANDATORY: Verify tool availability before selecting a branch.** > Do NOT assume answers — run the following commands in the terminal and use the actual output to choose the correct branch: > ```bash > # Unix/macOS — verify CLI exists AND daemon/runtime is functional > which docker && docker info > /dev/null 2>&1 && echo "Docker: available" || echo "Docker: NOT available" > which node && node --version && echo "Node.js: available" || echo "Node.js: NOT available" > ``` > ```powershell > # Windows (PowerShell) > Get-Command docker -ErrorAction SilentlyContinue | ForEach-Object { docker info >$null 2>&1; if ($?) { "Docker: available" } else { "Docker: NOT available (daemon not responding)" } } > Get-Command node -ErrorAction SilentlyContinue | ForEach-Object { node --version; if ($?) { "Node.js: available" } else { "Node.js: NOT available" } } > ``` > If a command succeeds, take the YES branch. If it fails (`command not found` or daemon not responding), take the NO branch. > ⚠️ `docker --version` is NOT sufficient — it succeeds even when the daemon is stopped. Use `docker info` instead. > Skipping this verification and defaulting to a fallback branch is a defect. ⚠️ **The two trees below are INDEPENDENT capability axes. Evaluate BOTH and combine the results.** Docker availability determines the **infrastructure tier** (Docker-based real infra vs embedded/in-memory alternatives). Node.js availability determines the **browser-E2E tier** (Playwright vs framework-native HTTP testing). Losing one capability does NOT affect the other axis. ``` Has Docker? ├── YES │ ├── Java/Spring → Testcontainers (DB, MQ, caches) │ ├── .NET / Node.js / Python / Go / Ruby → docker-compose or docker run (DB, MQ, caches) │ ├── Full stack with docker-compose? → docker-compose up + test against it │ └── Otherwise → start app + Docker-managed deps │ └── NO ├── Java/Spring → H2 in-memory DB + embedded alternatives for other infra (e.g., embedded Kafka, local mock services) — see §A.1 for H2 SQL compatibility risks ├── .NET → SQLite in-memory └── Node.js / Python / Go / Ruby → SQLite/in-memory store + mock services Has Node.js? ├── YES + App renders HTML (JSP, Thymeleaf, Razor, templates, SPA) │ ├── `npx playwright install --with-deps chromium` succeeds? │ │ └── YES → Playwright for browser E2E (catches view-layer bugs invisible to framework-native HTTP testing) │ │ └── NO → Framework-native HTTP testing fallback (document gap: browser rendering unverified; record exact install error) ├── YES + API-only (no HTML) │ └── Framework-native HTTP client (no Playwright needed) └── NO ├── Java with HTML views → framework integration test (e.g., MockMvc — covers cross-controller flows, forward/redirect, security; only gap is actual view rendering) ├── API-only (any language) → framework-native HTTP testing └── No Node.js + non-Java with HTML views → document gap, prioritize getting Node.js ``` #### Capability Prerequisites Beyond Docker and Node.js, verify these additional prerequisites when they apply: | Capability | Prerequisite | Verify Command | If Missing | |-----------|-------------|----------------|------------| | Docker-based infra | Docker daemon **functional** | `docker info > /dev/null 2>&1` | Embedded/in-memory alternatives (H2/SQLite for DB, embedded broker for MQ, local mocks) | | Playwright | Node.js + browser binaries | `npx playwright install --with-deps chromium` | MockMvc / TestRestTemplate | | Build tools | Maven or Gradle on PATH | `mvn --version` or `gradle --version` | Cannot proceed — escalate | | Headless browser | Playwright headless mode | Set `headless: true` in Playwright config | Always use headless in CI | | Network (external) | Outbound HTTPS for image pulls / browser downloads | `curl -s --max-time 5 https://registry.npmjs.org` | Use pre-cached binaries or mock external services | > These are secondary checks — run them after the Docker/Node.js primary axes. If any fails, document it as a constraint that may force fallback on that specific axis. ### 1.3.2 Legacy Test Asset Inventory Before finalising the testing strategy, identify the project's **canonical test command** — the top-level command used in CI (e.g., `yarn test:unit`, `mvn test`, `pnpm test`, `npm test`). Find it in the root `package.json` scripts, `pom.xml`, or `build.gradle`. Run it first and record the baseline passing count. All new tests written during migration must be reachable by this same command — do not introduce a separate test tool that bypasses it. If a new framework is added (e.g., vitest alongside Jest), it must be wired into the canonical command so both run together. Then check whether legacy E2E or integration tests are available. They may come from two sources — check both: **Source 1 — User-provided tests** The user may directly supply test files or paste test code in their request. These take priority over anything discovered on disk. Accept them as-is and skip the file-scan for the journeys they already cover. **Source 2 — Tests discovered in the source project** Scan the source project for existing test files. This is the fallback when the user provides nothing. **Inventory procedure:** 1. If the user explicitly provided test files or code, record them as `legacyTestAssets` with source `user-provided` and skip to step 3 2. Otherwise, locate existing test files in the source project (e.g., `src/test/`, `e2e/`, `cypress/`, `tests/`) 3. Classify each test suite by framework: | Framework / Pattern | Classification | Migration Target | |---|---|---| | AssertJ + Selenium / HtmlUnit | Legacy Java browser E2E | Rewrite to Playwright | | RestAssured / MockMvc (`@SpringBootTest`) | Java integration / API tests | Keep as-is if stack unchanged; rewrite if REST contract changes | | Cypress | JS/TS browser E2E | Evaluate rewrite to Playwright (see §1.3.3) | | Playwright (existing) | Modern browser E2E | Reuse with minimal edits (see §1.3.3) | | JUnit / pytest / xUnit (unit tests only) | Unit tests | Out of scope — implementer's job | 3. Record findings in the Testing Strategy deliverable under `legacyTestAssets`, including source (`user-provided` or `discovered`) **Output rule:** If legacy E2E tests are found, the Testing Strategy MUST specify one of: - **rewrite** — test framework incompatible with target stack (e.g., Selenium → Angular SPA) - **reuse-with-edits** — test framework compatible; only failing selectors / URLs need updating (e.g., existing Playwright suite) - **discard** — tests are so tightly coupled to old UI structure that rewriting from scratch is faster; document why ### 1.3.3 Legacy Test Migration Strategy Apply the appropriate strategy based on test classification: **Case A — Rewrite (e.g., AssertJ/Selenium → Playwright)** When the tech stack changes significantly (e.g., server-rendered JSP → Angular SPA), legacy Java browser tests cannot be run against the new frontend. They must be rewritten: 1. Map each existing test class / method to a Playwright test by the user journey it covers (not line-by-line) 2. Preserve the semantic intent of each test — same journey, same pass/fail criteria 3. Generate new `*.spec.ts` Playwright tests in the target project's `e2e/` or `tests/` directory 4. Document in the Testing Strategy: `legacyTestMappingTable` — old class → new spec file, journey parity confirmed **Case B — Reuse with minimal edits (existing Playwright suite)** When the source already has Playwright tests and the same journeys are still valid after migration: 1. **Copy the existing test files into the new project first** — do not rewrite from scratch 2. Run the suite against the migrated app and collect failures 3. Apply the **minimum-diff principle**: only change what breaks — selectors, URLs, data-ids, auth sequences. Do NOT restructure passing tests. 4. Track all changes in a diff table in the test execution report: `| test file | reason changed | lines changed |` 5. If more than 50% of a test file requires changes, flag it as a candidate for rewrite instead (document reasoning) **Recording in Testing Strategy:** ```markdown ### Legacy Test Assets - legacyTestAssets: [e.g., "AssertJ/Selenium suite — 12 test classes in src/test/java/e2e/"] - source: user-provided | discovered - migrationDecision: rewrite | reuse-with-edits | discard - migrationRationale: [reason] - legacyTestMappingTable: [old class → new spec, or N/A if discard] ``` ### 1.4 Testing Strategy Deliverable The implementation plan MUST include a "Testing Strategy" section containing: 1. **appType** — API-only / server-rendered HTML / SPA / mixed / messaging / desktop 2. **Critical user journeys** (3-5 minimum) — identified from feature inventory 3. **primaryValidationStack** — the full stack to execute when all capabilities are available 4. **fallbackMatrix** — for EACH independent capability axis, the fallback when that axis's prerequisite is unavailable: - `infra-tier`: primary tool (e.g., Testcontainers for Java, docker-compose for others) → fallback (e.g., H2, SQLite, embedded broker, local mock). Prerequisite: Docker - `browser-tier`: primary tool (e.g., Playwright) → fallback (e.g., framework-native HTTP testing). Prerequisite: Node.js + `npx playwright install` 5. **Environment requirements** — Docker? Node? What containers? What browser tooling? (list per-capability) 6. **knownGaps** — per-axis coverage loss when each fallback is activated 7. **Test data strategy** — how to seed, isolate (UUID), and clean up test data 8. **Acceptance criteria** — what constitutes PASS for each journey 9. **Validation review expectations** — what the final reviewer should confirm about test process and evidence 10. **Unit/integration test infrastructure** — the scaffold that enables implementers to run tests during the execute phase. The testing strategy MUST specify: - **External dependency strategy**: for each external dependency (DB, message broker, cache, other services, third-party APIs, file systems, frontend-backend API contracts, etc.), specify whether to use a real instance (via containers / Dev Services / in-memory) or a mock/stub, and how to configure it. For cross-boundary dependencies that exchange data via protocols (REST, gRPC, GraphQL, WebSocket, etc.), define the contract testing or mock server approach so each side can be tested independently. Each module must be testable without requiring manual setup - **Test base classes**: shared setup (container lifecycle, test data seeding, auth helpers) - **Per-module test skeletons**: startup-verification tests for each service module — created during scaffold, filled by implementers - **Test execution rule**: every implementation task MUST run module-level tests and report pass/fail/skip counts. Test failure = task not complete. No deferral to validation phase. ### Default Decision Policy by Project Type | Project Type | Primary Validation Stack | |-------------|--------------------------| | Server-rendered web app (any language) | Playwright + framework integration tests + Docker-based infra | | SPA (React/Angular/Vue) | Playwright | | Pure REST API (any language) | Framework integration tests + Docker-based infra | | Mixed app (API + browser UI) | Playwright + framework integration tests | | Messaging / event-driven | Docker-based infra + framework test runner | ### Per-Capability Fallback Matrix ⚠️ **Fallbacks are per-capability axis, NOT per-stack.** Docker and Node.js are independent axes. Losing one capability does NOT affect the other. Apply each row independently based on which specific capability is actually unavailable. | Capability Unavailable | What Changes | What Stays Unchanged | Gap Introduced |