test-strategy-planner

name

test-strategy-planner

description

Decide what tests to write and how to run them for any implementation task, during planning, before code is written. Use this whenever you are planning, scoping, or designing work on an application, API, UI, library, CLI, database schema, data migration, ETL/pipeline, infrastructure-as-code, or cloud deployment — and you need to produce a test plan. Use it even when no dedicated test framework or tool is available: the point of this skill is that an agent can design, build, and run almost any test from general-purpose primitives (shell, the language's native test runner, the database client, the cloud CLI, `terraform show -json`) without specialized tooling. Trigger on phrases like "what should I test", "test plan", "test strategy", "how do I test this", "add tests", "test coverage for this task", or any planning step for a task that will ship.

Test Strategy Planner

What this skill is for

Produce a test plan during planning: a list of the specific tests a task needs, what each one verifies, and exactly how the agent will build and run it. The output is consumed by whatever harness is implementing the task (GitHub Copilot, Claude Code, Codex) so it writes the tests alongside the code and runs them as its own feedback signal.

This skill exists to correct one wrong belief: "I can't test this because I don't have a tool for it." That is almost never true. Read the next section before doing anything else — it is the reasoning the rest of the skill depends on.

The only mental model you need

Every test, in every domain, is three steps and one signal:

1. Arrange — get the system into a known starting state.
2. Act — perform the operation under test.
3. Assert — compare what happened against what was expected.
4. Signal — emit a pass/fail. The universal pass/fail signal is the process exit code: 0 = pass, non-zero = fail. CI and the agent both read this. A test that does not exit non-zero on failure is not a test.

The "assert" step is where juniors think they need a framework. They do not. An assertion is just: run something, capture its output and exit code, compare to expectation. curl returns a status code. psql returns rows. aws ... returns JSON. terraform show -json returns a plan you can inspect. The language's native runner (pytest, Pester, xUnit, Vitest) wraps this in nicer syntax, but the mechanism underneath is identical.

So the planning question is never "what tool tests this?" It is "what is the observable thing I can check, and what command reveals it?" If you can describe "done" as something observable, you can test it.

One caveat, so this isn't taken too far: when a native test runner exists for the language, prefer it over hand-rolled shell assertions. pytest over a pile of if [ ... ]; then exit 1; fi. Native runners give you readable failures, fixtures/teardown, parametrization, and parallelism for free. "You don't need to procure a tool" is not "never use the runner that already ships with your stack." Use ad-hoc shell only when no runner fits (e.g. asserting on a terraform plan or a deployed cloud resource from CI).

Planning procedure

When invoked during planning, work through these five steps and emit the Test Plan at the end.

1. Restate "done" as observable acceptance criteria

Turn the task into a list of checkable statements. Vague: "build the user endpoint." Observable: "GET /users/{id} returns 200 with a JSON body matching the user schema; returns 404 for an unknown id; returns 401 without a token." Each observable statement maps to at least one test. If you cannot phrase a criterion observably, that is a signal the requirement is underspecified — flag it rather than guessing.

2. Classify the domain(s)

A task often spans more than one. Pick all that apply and pull the relevant catalog file:

Domain	Reference file
App, HTTP/gRPC API, service, library, CLI	references/app-and-api.md
Database schema, migration, query, stored proc, data integrity	references/data-and-database.md
Infrastructure-as-code, cloud resources, deployments	references/infra-and-cloud.md
Build, container, CI/CD pipeline, dependency/security gates	references/build-and-pipeline.md
Browser UI flows	Use the Playwright MCP (already available)

Read only the files relevant to the task. Each lists the common agentic tests for that domain with a concrete, runnable example of how to build it from general tooling.

3. Size by blast radius and reversibility (risk tier)

Right-sizing matters as much as the test list. Over-testing a throwaway script wastes time; under-testing an irreversible production change is how outages happen. The driver is how much breaks if this is wrong, and how hard it is to undo — not just which domain it's in.

Tier	What it is	Test depth
T0	Throwaway, internal-only, trivially reversible (a one-off script, a spike)	Smoke test only — does it run and produce the expected shape of output
T1	A standard service/feature, reversible by redeploy	Unit + integration + happy/error path + a smoke check
T2	Shared or stateful — touches a shared DB, a published contract, other teams' consumers	T1 + contract tests + migration up/down + idempotency
T3	Production, regulated, or hard/impossible to reverse — IAM, prod data, money movement, schema on a live table	T2 + plan-level policy assertions + post-deploy verification + drift check + explicit rollback test

When in doubt, tier up one level. State the chosen tier in the plan and why.

4. Select tests from the catalog

For each acceptance criterion, choose the test type(s) that verify it, at the depth the tier requires. Prefer the cheapest test that gives real confidence: a unit test beats an integration test beats an end-to-end test when it can actually catch the failure. Don't write an E2E test for logic a unit test covers — but don't unit-test wiring that only an integration test exercises.

5. Emit the Test Plan

ALWAYS output in this template:

## Test Plan: <task name>

**Risk tier:** <T0–T3> — <one line on blast radius / reversibility>
**Domains:** <app / database / infra / ...>
**Runner(s):** <pytest / Pester / xUnit / Vitest / shell+CLI>, run via <command>

### Acceptance criteria → tests
| # | Observable criterion | Test type | Verifies by | Where it lives |
|---|---|---|---|---|
| 1 | GET /users/{id} returns 200 + valid schema | API integration | request endpoint, assert 200 + jsonschema | tests/test_users.py |
| 2 | Migration is reversible | DB up/down | apply + assert schema, rollback + assert clean | tests/test_migration.py |
| ... |

### Non-functional checks (include per tier)
- [ ] Idempotency: <re-run is safe>  (T2+)
- [ ] Policy/plan assertion: <e.g. storage_encrypted=true>  (T3)
- [ ] Post-deploy verification: <CLI describe asserts live config>  (T3)
- [ ] Drift check: re-plan shows no changes  (T3)

### How CI runs it
<the single command CI invokes; every test exits non-zero on failure>

### Open questions / underspecified criteria
<anything from step 1 that couldn't be phrased observably>

Guardrails for the tests you recommend

These keep agent-written tests from becoming flaky liabilities:

• Fail loudly. Non-zero exit on any failure, always. A test that prints "FAILED" and exits 0 silently passes CI.
• Deterministic. No dependence on wall-clock timing, network flakiness, or ordering. Never sleep then assert — poll for the condition with a timeout instead (see the container example in build-and-pipeline.md).
• Self-cleaning. Integration and infra tests must restore state — drop the scratch DB, destroy the ephemeral stack, remove the container. Use the runner's teardown/fixture mechanism.
• Isolated from production. Never seed from, assert against, or write to production data or accounts. Use ephemeral resources (a docker run Postgres, a temp DuckDB file, a sandbox cloud account, a -targeted scratch stack).
• Test your config, not the vendor. Don't test that AWS creates an S3 bucket — Amazon tests that. Test that your Terraform requests encryption, your security group denies 0.0.0.0/0, your wiring connects the app to the right DB.
• Idempotent setup. Agents re-run things constantly. Setup and migrations must be safe to run twice.

Important: a recommendation is not a control

This skill produces recommendations during planning. Recommendations get skipped under deadline. If the goal is an enforced standard, the test plan has to be backed by a teeth mechanism — a required GitHub status check that runs the plan's CI command, a coverage gate, or a PR template that lists the plan's criteria as a checklist reviewers must confirm. When emitting a plan for a T2/T3 task, state plainly that it should be wired to a required check, not left to goodwill.