| name | e2e-testing-strategy |
| description | Decide when end-to-end tests pay off, how to budget flake, and how to split smoke vs full regression suites. Use when asked "should we add an E2E test for this?" or when an E2E suite has grown too slow or too flaky to trust. |
End-to-End Testing Strategy
Instructions
E2E tests exercise the whole app against real (or near-real) backends on a real device. They are the most realistic and the most expensive layer: slow, flaky-prone, and high-maintenance. Keep them small, keep them critical, and budget their flake explicitly.
1. What E2E Is For
E2E is uniquely good at catching:
- Wiring bugs — DI, routing, manifest/Info.plist, permissions, deep links.
- Build-specific bugs — ProGuard / R8 stripping, Bitcode, iOS entitlements, App Transport Security.
- First-launch flows — onboarding, auth, splash to home.
- Release-gate regressions — store checkout, payment, push notification delivery.
It is a bad tool for asserting logic edge cases, field validation, or API response shapes. Those belong in unit, integration, and contract tests.
2. Decide: Do We Need an E2E Test?
Ask these questions in order. Add the test only when the answer to all is yes:
- Does this flow fail in ways that unit, integration, UI-component, and contract tests cannot catch?
- Is a failure in this flow a release-blocker (revenue, login, safety, legal)?
- Will the test remain stable when unrelated features change around it?
If the answer is no to any, cover it at a cheaper layer.
3. Smoke vs Regression
Split the E2E suite into two tiers with different cadences:
- Smoke (< 10 min total, runs on every PR): launch, login, open main tabs, one happy-path purchase. If smoke fails, the PR fails.
- Regression (runs on merge queue or nightly): broader flows, edge cases, multi-step scenarios. Flake here triggers an investigation ticket, not a PR block.
Flag tests with @Smoke / @Regression and filter in CI via tags (--filter, -DincludeTags=Smoke, Maestro tags:).
4. Flake Budget
A suite of 100 E2E tests each 99 % reliable passes end-to-end only 37 % of the time. Two options:
- Keep reliability per-test ≥ 99.5 % (preferred; see the
flaky-test-reduction skill).
- Cap suite size — prefer 10 great tests over 100 mediocre ones.
Set an explicit flake budget, for example: "any test flaking more than 0.5 % over the last 100 runs is quarantined within 24 h". Track it in your CI dashboard.
5. Hermetic-ish Environment
Real "end-to-end" is a spectrum. Move the dial left (more hermetic) where possible:
- Seeded backend in a dedicated environment (not shared staging). A test harness that creates a user and seed data per run is worth every hour invested.
- Feature flags pinned to known values for the duration of the run.
- Payment providers in sandbox with deterministic card numbers.
- Time-sensitive flows — allow the harness to bump server time.
6. Test Data Lifecycle
- Create data at the start of the test via API, not through UI.
- Tear down data at the end, or use short-lived per-run users (
e2e+$runId@example.com).
- Never rely on production-like data that "usually exists".
7. Device & OS Coverage
- Run smoke on one device per platform per PR.
- Run regression across min-supported OS + latest OS + one low-end device nightly on a device farm.
- Do not run E2E on every OS/device combination — combinatorics explode and you will learn nothing new after the first few.
8. Reporting and Ownership
- Every E2E failure must surface: screenshot, video, device logs, network HAR (where available), and the seeded user/account ID.
- Each E2E test has a single owning team listed in the test file's header. Unowned E2E tests are deleted.
9. When to Delete an E2E Test
Be aggressive. Delete an E2E test when:
- Its failure mode is now covered by a unit or contract test.
- It has been quarantined for 30+ days without a fix.
- It asserts on cosmetic details that no longer match the design system.
Keeping dead E2E tests around erodes trust in the suite as a whole.
10. Checklist
