core-verify-testing-strategy

name	core-verify-testing-strategy
description	Use before writing or revising tests to choose the smallest useful evidence for each behavior — balance risk, cost, speed, maintainability, utilization, reliability, and fidelity; avoid pyramid cargo cults, coverage gaming, and flaky signals; record gates, feedback, and out-of-scope claims in any language or runtime.

Testing Strategy

Testing strategy decides what evidence a system needs and where that evidence should come from. It is not a target number of tests or a fixed shape on a pyramid. A good strategy makes the important failures visible quickly and keeps the signal trustworthy over time.

Start with claims and risk

Before adding a suite or expanding a test layer, write down the claims the release must support. For each claim:

1. Name the externally observable behavior, contract, or quality property.
2. List credible failure modes and the impact if each reaches a user or another system.
3. Identify changed code paths, trust boundaries, state transitions, and important operating conditions.
4. Estimate the value of earlier detection against implementation, maintenance, compute, environment, and human investigation costs.
5. Choose evidence that can actually observe the failure, then record what remains unproven.

Use a small planning table when the scope is not obvious:

| Claim or behavior | Failure impact | Evidence | Gate | Known gap |
|-------------------|----------------|----------|------|-----------|

Risk is not only severity. Include likelihood, change frequency, uncertainty, reversibility, and how long a bad result can remain hidden. Critical paths, authorization, calculations, migrations, data loss, concurrency, recovery, and regressions from known bugs usually need explicit proof.

Choose layers by evidence properties

No layer is inherently best. Choose the cheapest layer that proves the claim with enough fidelity. Evaluate each candidate by five properties:

• Speed — how soon it gives useful feedback.
• Maintainability — the cost of understanding, debugging, and updating it.
• Utilization — its use of CPU, memory, disk, devices, services, and other scarce resources.
• Reliability — whether a failure is a dependable signal of a problem.
• Fidelity — how closely it reflects the real operating conditions.

The trade-off is often remembered as SMURF: Speed, Maintainability, Utilization, Reliability, and Fidelity. Use the properties to choose deliberately; do not optimize one number while making the suite slow, opaque, or untrustworthy.

Evidence layer	Best for	Main trade-off
Component or unit	Deterministic rules, transformations, parsing, state transitions, and edge cases at a public seam	Fast and precise, but may miss wiring and environmental behavior
Integration	Real collaboration across a database, filesystem, queue, service adapter, or framework boundary	Higher fidelity, with more setup and resource cost
Contract	Public API, event, schema, or component expectations between independent consumers and producers	Finds drift at a boundary, but does not prove the whole journey
System or critical journey	A small set of user-visible flows and irreversible actions under realistic conditions	High fidelity, but slower and harder to diagnose
Exploratory or manual	Unknown risks, human judgment, rendering, unusual environments, and new hypotheses	Finds different failures, but is less repeatable and harder to gate
Specialized proof	Accessibility, performance, security, concurrency, resilience, localization, or model behavior	Proves a specific property; do not substitute general tests for it

The distribution of tests is an outcome of the claims and their costs. A pyramid is a useful starting heuristic, not a policy. A test hourglass — many component tests and many system tests with little boundary coverage — is a prompt to improve the architecture or add reliable integration evidence, not a reason to add more end-to-end tests blindly.

Use minimal, explicit fixtures for deterministic rules and representative, privacy-safe data for integration, migration, serialization, locale, and system risks. Do not let shared magical data or a convenient environment make a claim look proven when it is not.

Put evidence at the right gates

Place feedback where it helps the decision:

• During development: run fast, deterministic checks close to the change.
• At the change gate: block on the agreed checks for the changed behavior and its contracts.
• After integration: run broader suites, slower environments, and cross-system checks.
• Before release: exercise critical journeys, recovery, operational limits, and any risk the earlier layers cannot model.
• After release: compare field failures, support reports, performance, and real usage with what the pre-release evidence predicted.

The strategy names the owner, environment, data source, timeout, and diagnostic output for each gate. Keep feedback early enough that a failure still has a small search space.

Use coverage as a map, not a verdict

Coverage reports show which code or conditions were exercised. They do not show whether the assertions were meaningful, whether the expected value was independent, or whether the tested environment matches production. Use coverage to find untested changed or risky areas and to track trends. Do not turn one percentage into proof of quality or encourage tests that only execute lines.

When the cost is justified, use mutation testing or an equivalent fault-injection check to ask whether the suite detects small, plausible faults. Treat surviving mutants as investigation leads, not as a universal score or an excuse to add assertions without a behavior to prove.

Design for a trustworthy signal

A test failure should narrow the search for a real problem. Track and act on:

• nondeterministic results with unchanged code;
• failures caused by test data, setup, cleanup, ordering, time, randomness, or shared state;
• runner, operating-system, hardware, network, or dependency instability;
• time to diagnose and repair a failure;
• tests that never fail when their behavior is deliberately broken;
• runtime, resource use, and maintenance cost.

Classify a flaky result before rerunning it. Fix the source — isolate state, control time and randomness, synchronize on observable conditions, make environments reproducible, or repair the system under test. A retry may reduce noise while a fix is in progress, but it is not a quality strategy. Quarantine and ownership are explicit if a gate cannot yet be trusted.

Keep the strategy live

Write down why each non-obvious layer exists, what it does not cover, and which gate runs it. After a production defect, ask which claim was missing, which evidence was too weak or too slow, and whether the test would have caught the actual failure. Remove tests whose cost exceeds their current value when no critical risk depends on them. Keep the resulting gap visible with a manual, monitoring, exploratory, or follow-up action.

The strategy is complete when every acceptance criterion has a named verification method, every critical risk has evidence or an explicit exception, the gates have owners and environments, and the remaining unknowns are recorded.

For the source crawl and thematic notes behind this synthesis, read upstream-google-testing-blog.md.