run-device-tests

# Run Device Tests Skill Build and run .NET MAUI device tests locally on iOS simulators, MacCatalyst, Android emulators, or Windows. ## Platform Support | Host OS | Supported Platforms | |---------|---------------------| | macOS | ios, maccatalyst, android | | Windows | android, windows | ## Tools Required This skill uses `bash` together with `pwsh` (PowerShell 7+) to run the PowerShell scripts. Requires: - `xharness` - Global dotnet tool for running tests on iOS/MacCatalyst/Android - `dotnet` - .NET SDK with platform workloads installed - **iOS/MacCatalyst**: Xcode with iOS simulators - **Android**: Android SDK with emulator - **Windows**: Windows SDK ## Dependencies This skill uses shared infrastructure scripts: - `.github/scripts/shared/Start-Emulator.ps1` - Detects and boots iOS simulators / Android emulators - `.github/scripts/shared/shared-utils.ps1` - Common utility functions These are automatically loaded by the Run-DeviceTests.ps1 script. ## When to Use - User wants to run device tests locally - User wants to verify iOS/MacCatalyst/Android/Windows compatibility - User wants to test on a specific iOS version (e.g., iOS 26) - User asks "run device tests for Controls/Core/Essentials/Graphics" - User asks "test on iOS simulator" or "test on Android emulator" - User asks "run device tests on MacCatalyst" - User wants to run only specific test categories (e.g., "run Button tests") ## Available Test Projects | Project | Path | |---------|------| | Controls | `src/Controls/tests/DeviceTests/Controls.DeviceTests.csproj` | | Core | `src/Core/tests/DeviceTests/Core.DeviceTests.csproj` | | Essentials | `src/Essentials/test/DeviceTests/Essentials.DeviceTests.csproj` | | Graphics | `src/Graphics/tests/DeviceTests/Graphics.DeviceTests.csproj` | | BlazorWebView | `src/BlazorWebView/tests/DeviceTests/MauiBlazorWebView.DeviceTests.csproj` | ## Scripts All scripts are in `.github/skills/run-device-tests/scripts/` ### Run Device Tests (Full Workflow) ```bash # Run Controls device tests on iOS simulator (default on macOS) pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios # Run Core device tests on MacCatalyst pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform maccatalyst # Run Controls device tests on Android emulator pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android # Run Controls device tests on Windows (default on Windows) pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform windows # Run on specific iOS version pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform ios -iOSVersion 26 # Run with test filter pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "Category=Button" # Run one exact Core test class on Windows pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform windows -IncludeClasses "Microsoft.Maui.DeviceTests.WindowHandlerTests" # Run other test projects pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Essentials -Platform android pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Graphics -Platform maccatalyst pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project BlazorWebView -Platform ios ``` ### Build Only (No Test Run) ```bash pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -BuildOnly ``` ### Force a Full Rebuild Use `-Rebuild` when source files changed after an earlier build in the same worktree, such as an A/B verification run: ```bash pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform windows -Rebuild ``` ### List Available Simulators/Emulators ```bash # iOS simulators xcrun simctl list devices available # Android emulators emulator -list-avds ``` ## Workflow 1. **Run tests**: `scripts/Run-DeviceTests.ps1 -Project <name> -Platform <platform>` automatically detects/boots device, builds, and runs tests 2. **Check results**: Look at the console output or `artifacts/log/` directory for detailed test results ## Output - Build artifacts: `artifacts/bin/<Project>.DeviceTests/<Configuration>/<tfm>/<rid>/` - Test logs: `artifacts/log/` - Test results summary is printed to console ## Prerequisites - `xharness` global tool: `dotnet tool install --global Microsoft.DotNet.XHarness.CLI` - .NET SDK with platform workloads - **iOS/MacCatalyst**: Xcode with simulators installed - **Android**: Android SDK with emulator configured - **Windows**: Windows SDK - For iOS 26: macOS Tahoe (26) with Xcode 26 ## Examples ```bash # Quick test run for Controls on iOS (default on macOS) pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios # Test on MacCatalyst pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform maccatalyst # Test on Android emulator (works on both macOS and Windows) pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android # Test on Windows (default on Windows) pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform windows # Test on iOS 26 specifically pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -iOSVersion 26 # Run only Button category tests on iOS pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "Category=Button" # Build for Android without running tests pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform android -BuildOnly ``` ## Notes - The script automatically detects and boots an iOS simulator / Android emulator using the shared Start-Emulator.ps1 infrastructure - Default iOS simulator is iPhone Xs with iOS 18.5 (same as UI tests) - Default Android emulator priority: API 30 Nexus > API 30 > Nexus > First available - MacCatalyst runs directly on the Mac (no simulator needed) - Windows tests run directly on the local machine - Simulator/emulator selection and boot logic is handled by `.github/scripts/shared/Start-Emulator.ps1` - xharness manages test execution and reporting for iOS/MacCatalyst/Android - Class-filtered XHarness retries use isolated child output directories; Android also uses a per-run result filename so stale diagnostics or device-side XML cannot be reused - Windows runs the built device-test app directly and reads its xUnit XML results, matching `eng/devices/windows.cake` ## Test Filtering The `-TestFilter` parameter allows running specific test categories instead of all tests. This is useful for quick iteration during development. ### Filter Syntax | Format | Description | Example | |--------|-------------|---------| | `Category=X` | Run only category X | `Category=Button` | | `SkipCategories=X,Y,Z` | Skip categories X, Y, Z | `SkipCategories=Shell,CollectionView` | ### Examples ```bash # Run only Button tests on iOS pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "Category=Button" # Run only Button tests on Android pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android -TestFilter "Category=Button" # Skip heavy test categories pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "SkipCategories=Shell,CollectionView,HybridWebView" ``` ### How Test Filtering Works Test filtering is implemented in `src/Core/tests/DeviceTests.Shared/DeviceTestSharedHelpers.cs`: | Platform | How Filter is Passed | How Filter is Read | |----------|---------------------|-------------------| | **iOS/MacCatalyst** | `--set-env=TestFilter=...` | `NSProcessInfo.ProcessInfo.Environment["TestFilter"]` | | **Android** | `--arg TestFilter=...` | `MauiTestInstrumentation.Current.Arguments.GetString("TestFilter")` | | **Windows Controls** | App argument selects discovered category index | `ControlsHeadlessTestRunner` category loop | | **Windows non-Controls class filter** | Per-child `NUNIT_SKIPPED_CLASSES` plus the normal runner | XHarness `ApplicationOptions` class include | The Copilot Gate combines `-TestFilter` with `-IncludeClasses` when it knows the exact test class. On Windows, Controls still requires category discovery, while Core, Essentials, Graphics, and BlazorWebView bypass discovery and use XHarness's native class include. The result parser rejects output containing any unrelated class. Exact-class Windows runs, including scoped Controls category runs, are capped at 10 minutes per attempt; the Gate retries a timeout three times before treating repeated target-only timeouts as deterministic evidence. ### Available Test Categories Common categories in Controls.DeviceTests: - `Button`, `Label`, `Entry`, `Editor` - Individual control tests - `CollectionView`, `ListView`, `CarouselView` - Collection controls (heavy) - `Shell`, `Navigation`, `TabbedPage` - Navigation tests (heavy) - `Layout`, `FlexLayout` - Layout tests - `Memory` - Memory leak tests - `Accessibility` - Accessibility tests - `Gesture` - Gesture recognition tests To see all categories, check `src/Controls/tests/DeviceTests/TestCategory.cs`. ## Troubleshooting ### Xcode Version Mismatch If you see errors about Xcode version mismatch (e.g., "Xcode 26.2 installed but 26.0 required"): ```bash # Use -SkipXcodeVersionCheck to bypass validation pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -SkipXcodeVersionCheck ``` ### MacCatalyst App Bundle Not Found MacCatalyst apps use display names (e.g., "Controls Tests.app") not assembly names. The script handles this automatically by searching for `.app` bundles in the output directory. ### Android Package Name Issues Android package names must be lowercase (e.g., `com.microsoft.maui.controls.devicetests`). The script has a mapping of project names to correct package names. ### Test Filter Not Working 1. **Ensure full rebuild**: The test filter code must be compiled into the app. Delete artifacts and rebuild: ```bash rm -rf artifacts/bin/<Project>.DeviceTests pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android -TestFilter "Category=Button" ``` 2. **Check the console output**: Look for `TestFilter: Category=Button` in the test output to confirm filter was applied. 3. **Verify category exists**: Ensure the category name matches exactly (case-sensitive). ## XHarness Device Detection The script automatically handles XHarness device targeting for iOS and Android: ### iOS 1. **Simulator Boot**: Start-Emulator.ps1 detects and boots appropriate iOS simulator 2. **UDID Extraction**: Script extracts simulator UDID from Start-Emulator.ps1 output 3. **iOS Version Detection**: Script queries `xcrun simctl` to get iOS version from booted simulator 4. **XHarness Targeting**: Passes both `--target ios-simulator-64_VERSION` and `--device UDID` to xharness for explicit targeting ### MacCatalyst - Runs directly on the Mac, no device targeting needed - XHarness uses `--target maccatalyst` ### Android 1. **Emulator Boot**: Start-Emulator.ps1 detects running device or boots emulator 2. **Device ID Extraction**: Script gets device ID (e.g., `emulator-5554`) 3. **XHarness Targeting**: Passes `--device-id` to xharness android command ### Windows - No device/emulator needed - Runs the built device-test app directly and parses `TestResults-*.xml` **Why both --target and --device for iOS?** - XHarness requires `--target ios-simulator-64` (or `ios-simulator-64_VERSION`) to specify platform type - Adding `--device UDID` explicitly tells xharness which simulator to use - This combination ensures reliable device selection even on ARM64 Macs where automatic detection can fail **Example xharness invocations:** ```bash # iOS with test filter dotnet xharness apple test \ --app path/to/app \ --target ios-simulator-64_18.5 \ --device 56AE278D-60F7-4892-9DE0-6341357CA068 \ -o artifacts/log \ --timeout 01:00:00 \ -v \ --set-env=TestFilter=Category=Button # MacCatalyst with test filter dotnet xharness apple test \ --app path/to/app \ --target maccatalyst \ -o artifacts/log \ --timeout 01:00:00 \ -v \ --set-env=TestFilter=Category=Button # Android with test filter dotnet xharness android test \ --app path/to/app.apk \ --package-name com.microsoft.maui.controls.devicetests \ --device-id emulator-5554 \ -o artifacts/log \ --timeout 01:00:00 \ -v \ --arg TestFilter=Category=Button ``` This ensures tests run on the correct device with proper version targeting and filtering.