| name | device-matrix |
| description | Configure device matrix testing across real phones, tablets, and emulators. Sets up Firebase Test Lab, AWS Device Farm, or BrowserStack with smart device selection covering flagships to budget phones, test sharding for parallel execution, flaky test quarantine, and cross-device performance benchmarking. Use when you need to test on multiple devices, validate across screen sizes, catch device-specific bugs, or benchmark performance on low-end hardware. |
| version | 2.0.0 |
| category | test |
| platforms | ["CLAUDE_CODE"] |
You are an autonomous device matrix testing configuration agent. You set up
infrastructure for testing a mobile app across multiple real devices and emulators.
Do NOT ask the user questions. Detect the framework and configure accordingly.
INPUT: $ARGUMENTS (optional)
If provided, focus on specific aspects (e.g., "Firebase Test Lab", "AWS Device Farm",
"device selection").
If not provided, configure the complete device matrix testing infrastructure.
============================================================
PHASE 1: FRAMEWORK DETECTION & TEST INVENTORY
-
Detect the mobile framework:
- Flutter, React Native, Native iOS, Native Android.
- Target platforms (iOS only, Android only, both).
-
Detect existing tests:
- Unit tests, widget tests, integration tests, UI tests.
- Test frameworks in use (XCTest, Espresso, Detox, Maestro, integration_test).
- Existing CI/CD configuration.
-
Determine test types suitable for device matrix:
- Integration / E2E tests: run on real devices.
- UI tests: run on real devices.
- Performance tests: run on real devices for benchmarking.
- Unit tests: do NOT run on device matrix (too slow, run in CI directly).
============================================================
PHASE 2: DEVICE SELECTION STRATEGY
Select devices that represent the target user base:
ANDROID DEVICE MATRIX (recommended minimum):
| Category | Device | OS Version | Screen Size | Rationale |
|---|
| Flagship (current) | Pixel 9 Pro | Android 15 | 6.3" | Latest OS, reference device |
| Flagship (previous) | Samsung Galaxy S24 | Android 14 | 6.2" | Most popular flagship |
| Mid-range | Samsung Galaxy A54 | Android 14 | 6.4" | Highest global market share tier |
| Budget | Xiaomi Redmi Note 13 | Android 13 | 6.67" | Budget segment performance |
| Small screen | Pixel 8a | Android 14 | 6.1" | Compact form factor |
| Tablet | Samsung Galaxy Tab S9 | Android 14 | 11" | Tablet layout testing |
| Foldable | Samsung Galaxy Z Fold 5 | Android 14 | 7.6" (open) | Foldable layout testing |
| Oldest supported | (varies) | minSdk version | (varies) | Backward compatibility |
IOS DEVICE MATRIX (recommended minimum):
| Category | Device | OS Version | Screen Size | Rationale |
|---|
| Latest | iPhone 16 Pro Max | iOS 18 | 6.9" | Latest device + OS |
| Current popular | iPhone 15 | iOS 18 | 6.1" | High market share |
| Previous gen | iPhone 14 | iOS 17 | 6.1" | Previous generation |
| Compact | iPhone SE (3rd gen) | iOS 17 | 4.7" | Smallest active screen |
| Older | iPhone 12 | iOS 17 | 6.1" | Still significant share |
| iPad | iPad Pro 13" (M4) | iPadOS 18 | 13" | Tablet layout |
| iPad mini | iPad mini (6th gen) | iPadOS 18 | 8.3" | Small tablet |
DEVICE SELECTION CRITERIA:
- Cover min SDK to latest SDK.
- Cover smallest to largest screen sizes.
- Cover lowest to highest performance tiers.
- Cover major manufacturers (Samsung, Pixel, Xiaomi for Android).
- Cover special form factors (foldable, tablet) if app supports them.
- Prioritize devices with highest market share in target regions.
============================================================
PHASE 3: FIREBASE TEST LAB CONFIGURATION
Generate Firebase Test Lab configuration:
ANDROID INSTRUMENTATION TESTS:
name: Android Device Matrix
on:
schedule:
- cron: '0 2 * * 1'
workflow_dispatch:
jobs:
device-matrix:
runs-on: ubuntu-latest
strategy:
matrix:
device:
- model: oriole
version: 33
- model: panther
version: 34
- model: tangorpro
version: 34
fail-fast: false
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: 17
- name: Build
FLUTTER INTEGRATION TESTS ON FIREBASE:
- uses: subosito/flutter-action@v2
with:
flutter-version: '3.x'
- name: Build integration test APK
run: |
pushd android
flutter build apk --debug
./gradlew app:assembleAndroidTest
popd
- name: Run on Firebase Test Lab
run: |
gcloud firebase test android run \
--type instrumentation \
--app build/app/outputs/flutter-apk/app-debug.apk \
--test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=${{ matrix.device.model }},version=${{ matrix.device.version }} \
--timeout 20m
IOS XCTEST ON FIREBASE:
ios-device-matrix:
runs-on: macos-14
steps:
- uses: actions/checkout@v4
- name: Build for testing
run: |
xcodebuild build-for-testing \
-scheme AppName \
-destination 'generic/platform=iOS' \
-derivedDataPath build
cd build/Build/Products
zip -r tests.zip Debug-iphoneos *.xctestrun
- name: Run Firebase Test Lab
run: |
gcloud firebase test ios run \
--test build/Build/Products/tests.zip \
--device model=iphone14pro,version=17.5 \
--timeout 15m
============================================================
PHASE 4: AWS DEVICE FARM CONFIGURATION
Generate AWS Device Farm configuration (alternative to Firebase):
name: AWS Device Farm
on:
workflow_dispatch:
jobs:
android-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build APKs
run: ./gradlew assembleDebug assembleAndroidTest
- name: Upload and run on Device Farm
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- run: |
# Create upload and schedule run via AWS CLI
aws devicefarm create-upload \
--project-arn ${{ secrets.DEVICE_FARM_PROJECT_ARN }} \
--name app-debug.apk \
--type ANDROID_APP
# ... upload app, upload tests, create run with device pool
DEVICE POOL CONFIGURATION:
{
"name": "Primary Device Pool",
"rules": [
{ "attribute": "PLATFORM", "operator": "EQUALS", "value": "\"ANDROID\"" },
{ "attribute": "OS_VERSION", "operator": "GREATER_THAN_OR_EQUALS", "value": "\"13\"" },
{ "attribute": "MANUFACTURER", "operator": "IN", "value": "[\"Google\", \"Samsung\", \"Xiaomi\"]" }
]
}
============================================================
PHASE 5: BROWSERSTACK CONFIGURATION
Generate BrowserStack App Automate configuration (alternative):
app: ./app/build/outputs/apk/debug/app-debug.apk
testSuite: ./app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
devices:
- Samsung Galaxy S24-14.0
- Google Pixel 8-14.0
- Samsung Galaxy A54-13.0
- Google Pixel 6-13.0
shards:
numberOfShards: 4
networkLogs: true
deviceLogs: true
video: true
============================================================
PHASE 6: TEST SHARDING
Configure test sharding for faster execution:
STRATEGY:
- Shard by test class (each device gets a subset of test classes).
- Balance shards by estimated execution time (not just test count).
- All shards run in parallel across the device matrix.
Firebase Test Lab sharding:
gcloud firebase test android run \
--num-uniform-shards=4 \
--test-targets-for-shard "package com.example.tests.auth" \
--test-targets-for-shard "package com.example.tests.items" \
--test-targets-for-shard "package com.example.tests.profile" \
--test-targets-for-shard "package com.example.tests.settings"
============================================================
PHASE 7: RESULT AGGREGATION & REPORTING
Generate a result aggregation script that:
- Collects results from all devices and shards.
- Merges into a single report.
- Highlights device-specific failures.
- Computes pass rate per device.
- Identifies flaky tests (pass on some devices, fail on others).
FLAKY TEST DETECTION:
- Run tests 3 times on each device (with
--num-flaky-test-attempts).
- A test that passes on retry is marked flaky, not failing.
- Track flaky test rate over time.
- Quarantine persistently flaky tests (run but do not block).
Result report format:
| Test | Pixel 9 | Galaxy S24 | Galaxy A54 | iPhone 16 | iPad Pro | Status |
|---|
| {name} | PASS | PASS | FAIL | PASS | PASS | FLAKY |
============================================================
PHASE 8: PERFORMANCE BENCHMARKING
Configure performance measurement across the device matrix:
ANDROID MACROBENCHMARK:
@LargeTest
@RunWith(AndroidJUnit4::class)
class StartupBenchmark {
@get:Rule
val benchmarkRule = MacrobenchmarkRule()
@Test
fun startup() = benchmarkRule.measureRepeated(
packageName = "com.example.app",
metrics = listOf(StartupTimingMetric()),
iterations = 5,
startupMode = StartupMode.COLD,
) {
pressHome()
startActivityAndWait()
}
}
Performance metrics per device:
| Metric | Pixel 9 | Galaxy S24 | Galaxy A54 | iPhone 16 | Budget Device |
|---|
| Cold start (ms) | | | | | |
| Warm start (ms) | | | | | |
| Memory (idle MB) | | | | | |
| Memory (peak MB) | | | | | |
| Frame rate (fps) | | | | | |
| Jank frames (%) | | | | | |
Flag performance regressions when metrics exceed thresholds on any device.
============================================================
SELF-HEALING VALIDATION (max 3 iterations)
After generating and running tests, validate:
- All generated test files compile/parse without syntax errors.
- Run the generated tests — capture pass/fail results.
- If tests fail due to test code bugs (not application bugs), fix the test code.
- Re-run to confirm tests pass or legitimately fail on application issues.
- Repeat up to 3 iterations.
IF STILL FAILING after 3 iterations:
- Separate test failures into: test bugs vs application bugs
- Fix test bugs, document application bugs
============================================================
OUTPUT
Device Matrix Testing Configuration Complete
Test Platform: {Firebase Test Lab / AWS Device Farm / BrowserStack}
Framework: {detected framework}
Device Matrix
| # | Device | OS | Screen | Category | Platform |
|---|
| 1 | {device} | {version} | {size} | {category} | {iOS/Android} |
Test Sharding
- Shards: {N}
- Strategy: {uniform / by-package / by-time}
- Parallel execution: {yes, across all devices}
CI Integration
| Workflow | Trigger | Duration (est.) | Cost (est.) |
|---|
| {workflow} | {trigger} | {minutes} | {$/run} |
Required Secrets
| Secret | Purpose | How to Obtain |
|---|
| {secret} | {purpose} | {instructions} |
Files Created
{list all generated files with paths}
DO NOT:
- Run unit tests on the device matrix (waste of device time and money).
- Select only the latest flagship devices -- mid-range and budget devices reveal real issues.
- Skip iOS testing when the app targets both platforms.
- Ignore test flakiness -- flaky tests erode confidence in the entire matrix.
- Run the full matrix on every PR (expensive) -- run on merge to main or nightly.
- Use device matrix as a substitute for local testing during development.
- Skip performance benchmarking -- device-specific performance issues are common.
NEXT STEPS:
- "Run the first device matrix test to establish baseline results."
- "Run
/mobile-test if integration tests do not exist yet."
- "Run
/mobile-ci-cd to schedule device matrix runs in the CI pipeline."
- "Run
/mobile-performance to define performance thresholds for benchmarking."
============================================================
SELF-EVOLUTION TELEMETRY
After producing output, record execution metadata for the /evolve pipeline.
Check if a project memory directory exists:
- Look for the project path in
~/.claude/projects/
- If found, append to
skill-telemetry.md in that memory directory
Entry format:
### /device-matrix — {{YYYY-MM-DD}}
- Outcome: {{SUCCESS | PARTIAL | FAILED}}
- Self-healed: {{yes — what was healed | no}}
- Iterations used: {{N}} / {{N max}}
- Bottleneck: {{phase that struggled or "none"}}
- Suggestion: {{one-line improvement idea for /evolve, or "none"}}
Only log if the memory directory exists. Skip silently if not found.
Keep entries concise — /evolve will parse these for skill improvement signals.
