| name | test-automation-strategy |
| description | Expert guidance on scoping test automation investments, assigning ownership, choosing quality metrics, and keeping a mobile test suite economically viable. Use when asked "what should our test strategy be?" or when the test suite is becoming a cost center. |
Test Automation Strategy
Instructions
A test suite is an asset until it is not. Once maintenance cost exceeds the bugs caught, the suite is a liability. Strategy is about keeping the suite on the asset side of that line.
1. Start From the Business Risk
List the top risks for the app, ordered by impact and likelihood:
- Cannot sign in (all platforms).
- Cannot complete checkout / core action.
- Crashes on cold launch on minimum supported OS.
- Data loss on upgrade.
- Accessibility regressions that trigger legal exposure.
Every risk maps to exactly one test layer that should catch it first. Avoid covering the same risk at three layers — you pay three times and still get one signal.
2. Pyramid, Weighted by Cost and Signal
| Layer | Cost to Run | Cost to Maintain | Signal |
|---|
| Unit | Lowest | Lowest | Logic correctness |
| Integration | Low | Low-Medium | Wiring and serialization |
| UI component | Medium | Medium | Component behavior and a11y |
| Contract | Low | Low | Client/server mismatch |
| Snapshot | Low | Medium (review cost) | Visual regressions |
| E2E | Highest | Highest | End-to-end wiring, release-blockers |
Aim for ~70/20/10 (unit+integration / UI / E2E) by test count. Measure it; do not guess.
3. Ownership
Every test file has exactly one owning team, declared explicitly:
Unowned tests rot. When an owning team dissolves, the tests migrate or are deleted within one sprint.
4. Quality Metrics That Matter
Track these; publish them monthly:
- Escaped defect rate — bugs that reached production per release. The ultimate test-suite metric.
- PR feedback time — median time from PR open to CI complete. Under 15 min is aspirational; under 30 min is healthy.
- Suite flake rate — percentage of CI runs with at least one non-code-related failure.
- Coverage trend by package — not a single number. Target: trend up for new features, stable elsewhere.
- Test maintenance cost — % of PRs whose sole purpose is to fix or update tests.
Coverage as a gate number (e.g., "80 %") by itself is a weak signal. Track it, but do not fail builds on it.
5. Where to Invest Next
Use this triage loop each quarter:
- Identify the top 5 escaped defects of the last quarter.
- For each, ask: "what cheap test would have caught this?" Add or update that test.
- Identify the 5 flakiest tests; fix or delete them.
- Identify the 5 slowest tests; optimize or demote them to nightly.
- Delete any test that has not failed in 12 months and is not required for a risk above.
6. Scoping a New Feature
When a feature is planned, the automation plan accompanies the design:
- Which unit tests cover the pure logic?
- Which integration tests cover the seam to storage/network?
- Which UI tests cover the new screens (component level)?
- Which smoke or regression E2E flow, if any, is affected?
- Which a11y checks are required?
A feature without an automation plan is not ready to merge.
7. Test Code Quality
Tests are code and deserve the same review discipline:
- No duplicated setup bigger than a couple of lines — extract to builders / mothers.
- No commented-out tests. Delete or fix.
- No
@Ignore / xit without a linked ticket and a deadline.
- Lint test code with the same rules as production where possible.
8. Sunsetting Tests
Delete aggressively. Candidates:
- Duplicated by a cheaper layer.
- Always-green for 12+ months on a non-critical path.
- Quarantined for 30+ days.
- Tied to a feature that has been removed or hidden behind a killed flag.
Deleting tests is a positive act; the remaining suite becomes more trusted.
9. Communicating Strategy
Make these visible, not tribal:
- A one-page "how we test" doc linked from the README.
- A dashboard for flake, duration, and escaped defects.
- A standing slot in release retro to inspect the numbers.
10. Checklist
