run-integration-tests

# Run Integration Tests Skill Build the MAUI product, install local workloads, and run integration tests. ## When to Use - User asks to "run integration tests" - User asks to "test templates locally" - User asks to "validate MAUI build with templates" - User wants to verify changes don't break template scenarios - User asks to run specific test categories (WindowsTemplates, Samples, Build, Blazor, etc.) ## Available Test Categories | Category | Platform | Description | |----------|----------|-------------| | `Build` | All | Basic template build tests | | `WindowsTemplates` | Windows | Windows-specific template scenarios | | `macOSTemplates` | macOS | macOS-specific scenarios | | `Blazor` | All | Blazor hybrid templates | | `MultiProject` | All | Multi-project templates | | `Samples` | All | Sample project builds | | `AOT` | macOS | Native AOT compilation | | `RunOnAndroid` | macOS | Build, install, run on Android emulator | | `RunOniOS` | macOS | iOS simulator tests | ## Scripts All scripts are in `.github/skills/run-integration-tests/scripts/` ### Run Integration Tests (Full Workflow) ```powershell # Run with specific category pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "WindowsTemplates" # Run with Release configuration pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Samples" -Configuration "Release" # Run with custom test filter pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -TestFilter "FullyQualifiedName~BuildSample" # Skip build step (if already built) pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Build" -SkipBuild # macOS: Skip Xcode version check (for version mismatches) pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "macOSTemplates" -SkipBuild -SkipInstall -SkipXcodeVersionCheck # Auto-provision SDK if not found (first-time setup) pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Build" -AutoProvision ``` ## Parameters | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `-Category` | No | - | Test category to run (WindowsTemplates, Samples, Build, etc.) | | `-TestFilter` | No | - | Custom NUnit test filter expression | | `-Configuration` | No | Debug | Build configuration (Debug/Release) | | `-SkipBuild` | No | false | Skip build/pack step if already done | | `-SkipInstall` | No | false | Skip workload installation if already done | | `-SkipXcodeVersionCheck` | No | false | Skip Xcode version validation (macOS) | | `-AutoProvision` | No | false | Automatically provision local SDK if not found | | `-ResultsDirectory` | No | artifacts/integration-tests | Directory for test results | ## Workflow Steps The script performs these steps: 1. **Build & Pack**: `.\build.cmd -restore -pack -configuration $Configuration` 2. **Install Workloads**: `.dotnet\dotnet build .\src\DotNet\DotNet.csproj -t:Install -c $Configuration` 3. **Extract Version**: Reads MAUI_PACKAGE_VERSION from installed packs 4. **Run Tests**: `.dotnet\dotnet test ... -filter "Category=$Category"` ## Example Usage ```powershell # Run WindowsTemplates tests pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "WindowsTemplates" # Run Samples tests pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Samples" # Run multiple categories pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -TestFilter "Category=Build|Category=Blazor" ``` ## Prerequisites - Windows for WindowsTemplates, macOS for macOSTemplates/RunOniOS/RunOnAndroid - .NET SDK (version from global.json) - Sufficient disk space for build artifacts - Local SDK and workloads must be provisioned first ### Provisioning the Local SDK (Required First Time) Before running integration tests, you must provision the local .NET SDK and MAUI workloads: ```bash # Step 1: Restore dotnet tools dotnet tool restore # Step 2: Provision local SDK and install workloads (~5 minutes) dotnet cake --target=dotnet # Step 3: Install MAUI local workloads dotnet cake --target=dotnet-local-workloads ``` **Verification:** ```bash # Check SDK exists ls .dotnet/dotnet # Check MAUI SDK version ls .dotnet/packs/Microsoft.Maui.Sdk ``` > **Note:** The old `./build.sh --target=dotnet` syntax no longer works. Use `dotnet cake` directly. ## Output - Test results in TRX format at `<ResultsDirectory>/` - Build logs in `artifacts/` directory - Console output with test pass/fail summary ## Troubleshooting | Issue | Solution | |-------|----------| | "MAUI_PACKAGE_VERSION was not set" | Ensure build step completed successfully | | "Local .dotnet SDK not found" | Run `dotnet tool restore && dotnet cake --target=dotnet && dotnet cake --target=dotnet-local-workloads` | | Template not found | Workload installation may have failed | | Build failures | Check `artifacts/log/` for detailed build logs | | "Cannot proceed with locked .dotnet folder" | Kill processes using `.dotnet`: `Get-Process \| Where-Object { $_.Path -like "*\.dotnet\*" } \| ForEach-Object { Stop-Process -Id $_.Id -Force }` | | Session times out / becomes invalid | Integration tests are long-running (15-60+ min). Run manually in a terminal window instead of via Copilot CLI | | Tests take too long | Start with `Build` category (fastest), then run others. Use `-SkipBuild -SkipInstall` if workloads are already installed | | iOS tests fail with "mlaunch exited with 1" | Simulator state issue. Run individual tests instead of the whole category (see below) | | iOS simulator state errors (code 137/149) | Reset simulator: `xcrun simctl shutdown all && xcrun simctl erase all` or run tests individually | ## Running Manually (Recommended for Long-Running Tests) Integration tests can take 15-60+ minutes depending on the category. For best results, run directly in a terminal: ```powershell cd D:\repos\dotnet\maui # Option 1: Use the skill script pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Build" -SkipBuild -SkipInstall # Option 2: Run dotnet test directly (if workloads already installed) $env:MAUI_PACKAGE_VERSION = (Get-ChildItem .dotnet\packs\Microsoft.Maui.Sdk -Directory | Sort-Object Name -Descending | Select-Object -First 1).Name .dotnet\dotnet test src\TestUtils\src\Microsoft.Maui.IntegrationTests --filter "Category=Build" ``` ### Running All Categories Sequentially ```powershell # Windows categories (run on Windows) $categories = @("Build", "WindowsTemplates", "Blazor", "MultiProject", "Samples") foreach ($cat in $categories) { Write-Host "Running $cat..." -ForegroundColor Cyan pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category $cat -SkipBuild -SkipInstall } ``` ### Running Individual iOS Tests (Recommended) Running all iOS tests together (`-Category "RunOniOS"`) can cause simulator state issues. For better reliability, run tests individually: ```powershell # Available iOS tests $iosTests = @( "RunOniOS_MauiDebug", "RunOniOS_MauiRelease", "RunOniOS_MauiReleaseTrimFull", "RunOniOS_BlazorDebug", "RunOniOS_BlazorRelease", "RunOniOS_MauiNativeAOT" ) # Run a specific iOS test pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -TestFilter "FullyQualifiedName~RunOniOS_MauiDebug" -SkipBuild -SkipInstall -SkipXcodeVersionCheck # Run all iOS tests individually (more reliable than running category) foreach ($test in $iosTests) { Write-Host "Running $test..." -ForegroundColor Cyan pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -TestFilter "FullyQualifiedName~$test" -SkipBuild -SkipInstall -SkipXcodeVersionCheck } ```