| name | device-farms |
| description | Expert guidance on running mobile tests on Firebase Test Lab, AWS Device Farm, BrowserStack App Automate, and Sauce Labs. Use when asked to set up device-farm coverage, design a device matrix, or compare vendors. |
Device Farms
Instructions
Device farms provide breadth of OS and hardware coverage no local lab can match. The goal is not to run every test on every device — that is expensive and rarely informative — but to cover the shapes of risk (min OS, latest OS, one low-end device, one form-factor outlier).
1. Vendor Snapshot
| Vendor | Platforms | Strengths | Gotchas |
|---|
| Firebase Test Lab | Android (physical + virtual); iOS (XCTest via xctestrun) | Deep GCP integration, Robo crawler, generous free tier for OSS | iOS support is narrower; shell access limited |
| AWS Device Farm | Android + iOS | Appium, Espresso, XCUITest, flutter drive; per-minute pricing | UI is dated; artifact retention short by default |
| BrowserStack App Automate | Android + iOS (real devices) | Wide device matrix, fast session start, good iOS coverage | Interactive/manual sessions can compete for device pools |
| Sauce Labs Real Device Cloud | Android + iOS | Enterprise reporting, device reservations, Appium/XCUITest/Espresso | Pricing tier complexity |
| Maestro Cloud | Android + iOS | Native Maestro integration, auto-sharding | Only runs Maestro flows, not Espresso/XCUITest |
Pick one primary and one fallback. Multi-vendor abstraction layers rarely pay off — tests lose features across the lowest common denominator.
2. Device Matrix Policy
A good matrix is small and intentional:
- Android: min SDK + latest stable + 1 low-RAM device (e.g., Pixel 4a) + 1 tablet / foldable if relevant.
- iOS: min iOS supported + latest iOS + one iPhone SE (small screen) + one iPad if the app supports it.
Run smoke on a single device per platform per PR. Run the full matrix nightly or on release branches.
3. Firebase Test Lab (Android)
From CI:
gcloud firebase test android run \
--type instrumentation \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=redfin,version=33,orientation=portrait \
--device model=oriole,version=34,orientation=portrait \
--timeout 20m \
--use-orchestrator \
--num-flaky-test-attempts 1
Use --results-dir to pin artifact paths and --test-targets "class com.example.SmokeSuite" to shard.
For iOS XCTest on FTL, build a .xctestrun and .xcarchive, zip, and upload via gcloud firebase test ios run.
4. AWS Device Farm
aws devicefarm schedule-run \
--project-arn "$PROJECT_ARN" \
--app-arn "$APK_ARN" \
--device-pool-arn "$POOL_ARN" \
--name "pr-$CI_SHA" \
--test type=INSTRUMENTATION,testPackageArn="$TEST_APK_ARN"
Artifacts (video, logcat, test specs) are downloadable per run. Keep runs under 150 min; longer runs are aborted on some tiers.
5. BrowserStack App Automate — Example (Espresso)
curl -u "$BS_USER:$BS_KEY" \
-X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
-d '{
"app": "bs://APP_HASH",
"testSuite": "bs://TESTSUITE_HASH",
"devices": ["Google Pixel 8-14.0","Samsung Galaxy S24-14.0"],
"deviceLogs": true, "networkLogs": true, "video": true
}' -H "Content-Type: application/json"
BrowserStack's REST API is thin and CI-friendly. For Maestro, use their official GitHub Action.
6. Sauce Labs — RDC
Sauce uses a single saucectl config for multiple frameworks:
apiVersion: v1alpha
kind: espresso
sauce:
region: us-west-1
concurrency: 3
espresso:
app: ./app-debug.apk
testApp: ./app-debug-androidTest.apk
suites:
- name: smoke
devices:
- name: "Google Pixel 8"
platformVersions: ["14"]
7. Artifacts and Debugging
Every farm run should produce, at minimum:
- Video of the session.
- Logcat (Android) / syslog (iOS).
- Screenshots for each failing assertion.
- Test report in JUnit XML.
- Where supported, network HAR.
Fail the CI job loudly if artifacts are missing — "no repro" without artifacts is a process bug.
8. Cost Control
- Cap per-run duration (
--timeout, farm-specific).
- Shard big suites; short jobs run in parallel for less wall-clock.
- Use virtual devices on FTL for the bulk of runs; reserve physical devices for hardware-specific checks.
- Nightly full-matrix runs are cheaper per finding than full-matrix on every PR.
9. Flake on Farms
Device farms amplify any non-determinism in your tests. If a test flakes locally at 1 %, it will flake at 3–5 % on a shared pool. Fix locally first (see flaky-test-reduction) — never paper over farm flake with high retry counts.
10. Checklist
