testing-in-ci

name	testing-in-ci
description	Expert guidance on running mobile test suites in CI — matrix design, caching, sharding, artifacts, required checks — without paying for slow, flaky runs. Use when configuring GitHub Actions, GitLab CI, Bitrise, or Codemagic for mobile testing.

Testing in CI

Instructions

CI is where the test pyramid pays its rent or exposes its debts. Good CI is fast, predictable, and debuggable. The test suite that works on your laptop but takes 45 minutes on CI is failing its job.

1. Pipeline Tiers

Run different test layers at different cadences:

Trigger	What runs	Target wall-clock
Every commit on a PR	Unit + integration + lint + type-check	≤ 10 min
PR ready / merge queue	Above + UI component + smoke E2E	≤ 20 min
Nightly	Full regression + device-farm matrix + perf	≤ 2 h
Release branch	All of the above + manual QA gates	—

Do not run nightly-class tests on every PR. Engineers will ignore them.

2. Matrix Design — Small and Intentional

A common mistake: 3 JDKs × 4 API levels × 3 architectures = 36 jobs that never all pass together. Keep matrices small:

• Android unit: one JDK (the one the build uses), one NDK if applicable.
• Android instrumented: min SDK + latest SDK emulator.
• iOS unit: one Xcode version, one iOS simulator per min+latest.
• iOS UI: one iPhone + one iPad if supported.
• Flutter: one Dart/Flutter channel (stable); beta/master run nightly only.
• RN: one Node major that matches .nvmrc.

Document the matrix in the repo; revisit it quarterly.

3. Caching

Caching is the difference between 3-minute and 10-minute PR feedback.

• Gradle: cache ~/.gradle/caches, ~/.gradle/wrapper, .gradle/ per-project; key on *.gradle*, gradle-wrapper.properties, and gradle/libs.versions.toml.
• CocoaPods: cache Pods/ and ~/Library/Caches/CocoaPods; key on Podfile.lock.
• Swift Package Manager: cache .build/ and derived data per-package.
• Flutter: cache ~/.pub-cache and .dart_tool/ keyed on pubspec.lock.
• Node (RN): cache ~/.npm / ~/.yarn / pnpm store keyed on lockfile; also cache ios/Pods if using RN's autolinking.

Invalidate on lockfile changes; never cache across architectures.

4. Sharding

See parallel-test-execution for deep guidance. For CI in particular:

• Shard anything > 5 minutes on a single runner.
• Publish per-shard JUnit XML; merge at the end for PR summary.
• Balance shards by past durations (Gradle Enterprise, circleci tests split --split-by=timings, custom test-sharder).

5. Required Checks

Use required status checks (branch protection) sparingly:

• Lint, typecheck, unit, integration: required on PR.
• UI component: required if it is fast and stable; otherwise advisory.
• E2E smoke: required on the merge queue, not on every PR commit.
• Device-farm matrix, perf, mutation: advisory / nightly.

Every required check must have a p95 wall-clock and flake-rate SLA published; when a check breaks its SLA for two weeks, it becomes advisory until fixed.

6. Artifacts to Upload

For every test job:

• Always: JUnit XML (for PR summary), logs.
• On failure: screenshots, videos, logcat/os_log, HAR, HTML coverage, memory dumps.
• Device-farm runs: full run directory (~/.maestro/tests, xcresult, FTL results bucket).

GitHub Actions example:

- name: Upload test artifacts
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: ${{ matrix.shard }}-results
    path: |
      **/build/test-results/**/*.xml
      **/build/reports/**
      ~/.maestro/tests
    retention-days: 14

7. Build Determinism

• Pin toolchain versions: JDK, Xcode, Flutter, Node, Ruby (for CocoaPods / Fastlane).
• Use lockfiles (Gemfile.lock, Podfile.lock, pubspec.lock, package-lock.json/yarn.lock/pnpm-lock.yaml).
• Disable analytics / network dependency downloads mid-build where possible.
• Use hermetic emulators / simulators: cold boot from a known snapshot.

8. Secrets and Signing in CI

• Never commit signing keys. Load from the CI secret store.
• Use short-lived service credentials for device farms (OIDC to GCP / AWS where possible).
• Keep release signing jobs on a dedicated, slower runner with strict branch filters.

9. Observing the Pipeline

Track, per week:

• p50 / p95 PR feedback time.
• Flake rate per required check.
• Artifact download rate on failures (if low, artifacts are unhelpful — fix them).
• Cache hit rate (target > 80 %).

Publish a dashboard. Slow CI is a feature bug; measure it like one.

10. Common Pitfalls

• Emulator boot inside the test phase (counts against the test timeout).
• Running instrumented Android tests on every PR when a Robolectric suite would do.
• Over-parallelizing RN tests (Jest's --maxWorkers=100% contends with TypeScript check).
• Relying on "retry the whole pipeline" to get green; masks real flake.

11. Checklist

• Pipeline tiers (PR / merge queue / nightly / release) are defined.
• Matrices are minimal and documented.
• Lockfiles drive caches; cache hit rate > 80 %.
• Shards are balanced by duration; JUnit XML is merged for PR summary.
• Required checks each have a published p95 and flake SLA.
• Artifacts (logs, screenshots, videos) always upload on failure.
• Toolchain versions are pinned; lockfiles committed.
• Secrets come from CI vault, not files; OIDC preferred where supported.
• PR feedback p95 is tracked and defended.