| name | test-suite-setup |
| description | Complete end-to-end setup and execution of the WinUI3 test suite locally. Delegates building to the build skill, then handles test payload creation, machine setup, and running all tests. |
WinUI3 Full Test Suite Setup & Execution
This skill performs a complete end-to-end test run of the WinUI3 repository on the local machine.
It covers every step from environment initialization through running all ~8000 tests.
AI Agent Instructions
When something goes wrong, read these first. This skill has accumulated reference material later in this document that will save hours of debugging:
- HRESULT Decoder Table — every crash code we've seen (
0xC0000602, 0x8007007E, 0x80000003, etc.) with cause + first fix.
- OSS Package Coherence — symptom: cascading combase crashes / >10% External failures. Root cause is almost always an incoherent OSS-pinned package set, not your code.
- CI Test Invocation Map — what hosting mode each CI work item uses. Critical: Managed tests run with
/p:HostingMode=WPF, not AppX. Missing this flag will look like "all 700+ Managed tests blocked" when nothing is actually broken.
- Baseline Failure Catalog — methodology for "is this my bug or pre-existing?" before treating any failure as a regression.
- ADO ↔ GitHub Mirror Reference — which clone to edit when the file exists in both.
When this skill is invoked:
-
First, ensure the environment is ready — check if TestPayload exists and is populated. If not, invoke the build skill to initialize and build the repo (product + tests), then run Steps 3-5 below (create payload, machine setup, install runtimes).
Platform/configuration: The default flavor is x64 Debug (amd64chk). If the user requests a different architecture (e.g., arm64) or configuration (fre/release), pass the appropriate flags to the build skill and adjust -Platform/-Configuration in all subsequent steps. Supported platforms: x86, x64, arm64, arm64ec. Supported configurations: chk (debug), fre (release).
-
Determine the comparison branch, then analyze changed files to suggest relevant tests.
Branch detection logic (in order):
- If the user explicitly specified a branch (e.g., "compare against
release/2.8"), use that.
- Try to auto-detect the PR target branch:
git --no-pager config --get "branch.$(git branch --show-current).merge" 2>$null
- If the current branch name suggests a release/servicing target (matches
release/*, servicing/*, hotfix/*), ask the user via ask_user:
"Your branch appears to target a release branch. Which branch should I compare against?"
Choices: detected release branch, main, "Let me specify"
- Default to
main.
Get changed files:
git --no-pager diff --name-only <comparison-branch>...HEAD
If the comparison branch is not available locally, fetch it first:
git fetch origin <comparison-branch>
git --no-pager diff --name-only origin/<comparison-branch>...HEAD
If that also fails, fall back to:
git --no-pager diff --name-only HEAD~5..HEAD
-
Map changed files to test suites using a hybrid approach: known mappings first, then dynamic discovery for anything new, and a user fallback for unmatched files.
Step 3a: Known mappings (fast path)
These curated rules cover the most common source → test relationships:
| Changed files match | Recommended test DLLs | Hosting Mode |
|---|
dxaml/xcp/core/**, dxaml/xcp/dxaml/** | External.Foundation, External.Framework, External.Controls | WPF |
dxaml/xcp/components/imaging/** | Isolated.Foundation.Imaging | None |
dxaml/xcp/components/text/** | External.Foundation (text tests), Isolated.Text | WPF / None |
dxaml/xcp/components/focus/** | Isolated.Xaml.Focus.* | None |
dxaml/xcp/components/accesskeys/** | Isolated.AccessKeys.* | None |
dxaml/test/native/external/foundation/** | External.Foundation | WPF |
dxaml/test/native/external/framework/** | External.Framework | WPF |
dxaml/test/native/external/controls/** | External.Controls | WPF |
dxaml/test/native/external/automation/** | External.Automation | WPF |
dxaml/test/native/external/enterprise/** | External.Enterprise | WPF |
dxaml/test/native/external/win32/** | External.Win32 | WPF |
dxaml/test/native/external/activation/** | External.Activation |
-
Present the recommendation using ask_user. Show:
- The files that changed (summarized)
- The recommended test suites based on the mapping
- Choices:
Run recommended tests (Recommended), Choose different tests, Run all tests
If user picks "Choose different tests", dynamically list all available test DLLs from the TestPayload:
Get-ChildItem "<repo-root>\TestPayload\x64chk\Test\Microsoft.UI.Xaml.Tests.*.dll" |
ForEach-Object { $_.BaseName -replace '^Microsoft\.UI\.Xaml\.Tests\.', '' }
Present each discovered suite as a choice, plus:
All Tests (all hosting modes) — runs everythingCustom query — let user specify a wildcard pattern like *CommandBar*
Hosting mode inference for dynamically discovered DLLs:
- DLL name contains
External. → WPF
- DLL name contains
Isolated. → None
- DLL name contains
Managed. → WPF
MUXControls.Test.dll, UnitTests.dll → None- Unknown → ask user or try without hosting mode
-
Run the selected tests using te.exe directly for specific DLLs (more precise than runtests.cmd):
cd <repo-root>\TestPayload\x64chk
# For a specific test DLL (preferred for targeted runs):
.\te.exe Test\Microsoft.UI.Xaml.Tests.Isolated.Foundation.Imaging.dll /p:SkipConsoleWindowMinimize
# For External.* with WPF hosting:
.\te.exe Test\Microsoft.UI.Xaml.Tests.External.Controls.dll /p:SkipConsoleWindowMinimize /p:HostingMode=WPF
# For Managed.* with WPF hosting:
.\te.exe Test\Microsoft.UI.Xaml.Tests.Managed.Controls.dll /p:SkipConsoleWindowMinimize /p:HostingMode=WPF
# For multiple DLLs at once:
.\te.exe Test\Microsoft.UI.Xaml.Tests.Isolated.Foundation.Imaging.dll Test\Microsoft.UI.Xaml.Tests.External.Foundation.dll /p:SkipConsoleWindowMinimize
# Using runtests.cmd with wildcard (runs across all DLLs):
.\runtests.cmd *AnimatedImage* -HostingMode:WPF
# For ALL tests — run each hosting mode separately:
.\runtests.cmd * -HostingMode:UAP
.\runtests.cmd * -HostingMode:WPF
.\runtests.cmd * -HostingMode:Win32Explicit
.\runtests.cmd * -HostingMode:None
-
After tests complete, provide a detailed results summary. Parse the TAEF output and present:
Required summary format:
- A results table per test suite with columns: Total, Passed, Failed, Blocked, Skipped
- For each failed test: the full test name, the failure reason (e.g., assertion message, HRESULT, CRC mismatch), and the source file/line if available
- For each blocked test: the full test name and the blocking reason (e.g., missing privileges, missing Te.Service, test marked Ignore)
- An overall verdict: whether the changes are safe (no new failures) or need attention
- Actionable next steps for any failures or blocked tests (e.g., "run from admin prompt", "update master image", "pre-existing failure unrelated to your changes")
How to extract failure details: Search the TAEF output for lines containing [Failed], [Blocked], Error:, Verify: (failed assertions), and Summary:. Use Select-String or Select-Object -Last to find relevant lines. For each failed test, look for the lines between StartGroup: <testname> and EndGroup: <testname> [Failed] to capture the full failure context.
Example summary:
### Test Results: Isolated.Foundation.Imaging
| Total | Passed | Failed | Blocked | Skipped |
|-------|--------|--------|---------|---------|
| 39 | 38 | 1 | 0 | 0 |
#### Failed Tests
1. **DecodingUnitTests::Jpeg**
- Reason: CRC mismatch (Expected 0x5fd3ea9b, Actual 0xda5a7f30)
- Source: DecodingUnitTests.cpp, Line 111
- Action: Pre-existing environment-specific failure, unrelated to your changes. Can be ignored.
#### Blocked Tests
(None)
### Overall Verdict
✅ Your changes did not introduce any new test failures.
Classify failures as:
- New failure (likely caused by the current changes) — needs investigation
- Pre-existing failure (exists on main branch too) — note it but don't block
- Environment-specific (CRC mismatch, timing, machine config) — suggest rerun or admin prompt
If any tests are blocked due to missing admin privileges or Te.Service, provide the exact command to rerun them from an elevated prompt.
Hosting mode rules:
External.* tests → use /p:HostingMode=WPF for local runs (UAP requires full UWP container)Isolated.* tests → no hosting mode needed (unit tests, no UI)Managed.* tests → use /p:HostingMode=WPF (C# tests only work in WPF mode)MUXControls.Test.dll / UnitTests.dll → no hosting mode needed- When user selects "All Tests", run all 4 hosting modes sequentially
Prerequisites
- Windows 10/11 (20H2 or later recommended)
- Visual Studio 2022 with C++ workloads and Spectre mitigation libs
- If missing Spectre libs, import the
.vsconfig file from the repo root via VS Installer
- Admin privileges (required for test machine setup and running tests)
- A visible desktop session (tests launch UI apps and send input — do NOT run over minimized RDP or locked screen)
Step 1–2: Build the Repository (Delegate to Build Skill)
Before creating the test payload, the repo must be initialized and built (product + tests).
Invoke the build skill to handle this. When delegating, request:
- A full repo build (product + tests):
.\initrun.ps1 .\build.cmd /q /b
- The
/b flag is recommended for test-suite prep to avoid PCH memory exhaustion on limited-memory machines.
- For a clean first-time build, use
/c as well: .\initrun.ps1 .\build.cmd /q /c /b
The build skill handles init, flavor selection, troubleshooting (PCH errors, stale PCH, missing Spectre libs, etc.).
Note: amd64 and x64 are used interchangeably. init.ps1 uses amd64chk while CreateTestPayload.ps1 uses -Platform x64 -Configuration chk, but they produce output in the same TestPayload\x64chk\ directory. Ensure the build flavor matches the payload platform/configuration.
Step 3: Create the Test Payload
The test payload aggregates all built test binaries, TAEF infrastructure, and dependencies into a single directory.
.\initrun.ps1 powershell -ExecutionPolicy Bypass -File test\CreateTestPayload.ps1 -Platform x64 -Configuration chk
Parameters:
-Platform: x86, x64, arm64, arm64ec (must match your build flavor)-Configuration: chk or fre (must match your build flavor)-Mode: DevTestSuite (default, for functional tests) or ScenarioTestSuite (for sample app tests)-Clean: Wipe the TestPayload directory before creating
The payload is created at <repo-root>\TestPayload\<platform><config>\ (e.g., TestPayload\x64chk\).
Step 4: One-Time Machine Setup (Admin Required)
This sets up crash dump collection, installs test certificates, installs required AppX framework packages,
and kills processes known to interfere with tests.
# Open an ADMIN PowerShell prompt, then:
cd <repo-root>\TestPayload\x64chk
.\testmachine-prerun.cmd
What it does:
- Enables developer mode (sideloading without dev license)
- Disables network flyout
- Adds Windows Defender exclusion for test directory
- Kills interfering processes (Photos, Skype, YourPhone, etc.)
- Installs VCLibs and .NET runtime AppX packages
- Configures crash dump collection for
te.exe and te.processhost.exe
This only needs to be run once per machine (or after OS reinstall).
Step 5: Install .NET Desktop Runtime (if needed)
If managed (C#) tests fail to launch, install the .NET Desktop Runtime:
cd <repo-root>\TestPayload\x64chk
.\dotnet-windowsdesktop-runtime-installer.exe /install /quiet /norestart
Step 6: Install VC++ Redistributable (if needed)
cd <repo-root>\TestPayload\x64chk
.\vc_redist.x64.exe /install /quiet /norestart
Step 7: Run All Tests
From the TestPayload directory, use runtests.cmd to run tests. Tests use the TAEF (te.exe) framework.
Run ALL tests (all hosting modes):
WinUI tests run in different hosting modes (UAP, WPF, Win32Explicit). You need to run each mode separately.
cd <repo-root>\TestPayload\x64chk
# Run all UAP-hosted tests (native C++ tests):
.\runtests.cmd * -HostingMode:UAP
# Run all WPF-hosted tests (native + managed C# tests):
.\runtests.cmd * -HostingMode:WPF
# Run all Win32Explicit-hosted tests:
.\runtests.cmd * -HostingMode:Win32Explicit
# Run tests with no hosting mode (unit tests, app tests):
.\runtests.cmd * -HostingMode:None
Run a specific test or subset:
# Run tests matching a name pattern:
.\runtests.cmd *CommandBar*
# Run a specific test class:
.\runtests.cmd *CommandBarAutomationIntegrationTests*
# Run with WPF hosting:
.\runtests.cmd *MenuFlyout* -HostingMode:WPF
Useful flags:
# See test statistics without running:
.\runtests.cmd * -Stat
# List tests without running them:
.\runtests.cmd *CommandBar* /list
# Wait for debugger attachment:
.\runtests.cmd *MyTest* -WaitForDebugger
# Run including disabled/ignored tests:
.\runtests.cmd *MyTest* -RunIgnoredTests
# Run tests in a loop (for reliability testing):
.\runtests.cmd *MyTest* -RunTestsInALoop
# Stop on first failure:
.\runtests.cmd *MyTest* -TerminateOnFirstFailure
# Slow down test execution to watch it:
.\runtests.cmd *MyTest* /p:GoSlow
Using te.exe directly (advanced):
cd <repo-root>\TestPayload\x64chk
# Run a specific test DLL with WPF hosting:
.\te.exe Test\Microsoft.UI.Xaml.Tests.External.Controls.dll /p:SkipConsoleWindowMinimize /p:HostingMode=WPF
# Run with a TAEF select query:
.\te.exe Test\Microsoft.UI.Xaml.Tests.External.Controls.dll /select:"@Name='*CommandBar*'" /p:SkipConsoleWindowMinimize
# Run managed tests in WPF mode:
.\te.exe Test\Microsoft.UI.Xaml.Tests.Managed.Controls.dll /p:HostingMode=WPF /p:SkipConsoleWindowMinimize
Full Automated Script (All Steps)
Below is a complete PowerShell script that runs all steps end-to-end.
Must be run from an admin PowerShell prompt.
Adjust $platform and $config if targeting a different architecture (e.g., arm64 + chk for ARM64 debug).
# === WinUI3 Full Test Suite — End-to-End ===
# Run from admin PowerShell at repo root
$ErrorActionPreference = "Stop"
$repoRoot = Get-Location
# --- Configuration ---
# Change these to target a different architecture/configuration
$platform = "x64" # x86, x64, arm64, arm64ec
$config = "chk" # chk (debug) or fre (release)
$flavor = "${platform}${config}" # e.g., x64chk, arm64chk
# --- Step 1: Init ---
Write-Host "=== Step 1: Initializing build environment ===" -ForegroundColor Cyan
.\init.ps1
# --- Step 2: Build (clean, reduced parallelism) ---
Write-Host "=== Step 2: Building repository (product + tests) ===" -ForegroundColor Cyan
.\initrun.ps1 .\build.cmd /q /c /b
# --- Step 3: Create Test Payload ---
Write-Host "=== Step 3: Creating test payload ===" -ForegroundColor Cyan
.\initrun.ps1 powershell -ExecutionPolicy Bypass -File test\CreateTestPayload.ps1 -Platform $platform -Configuration $config -Clean
# --- Step 4: Machine Setup ---
Write-Host "=== Step 4: One-time machine setup ===" -ForegroundColor Cyan
Push-Location "$repoRoot\TestPayload\$flavor"
.\testmachine-prerun.cmd
# --- Step 5: Install runtimes if present ---
Write-Host "=== Step 5: Installing runtimes ===" -ForegroundColor Cyan
if (Test-Path ".\dotnet-windowsdesktop-runtime-installer.exe") {
Start-Process -Wait -FilePath ".\dotnet-windowsdesktop-runtime-installer.exe" -ArgumentList "/install /quiet /norestart"
}
if (Test-Path ".\vc_redist.x64.exe") {
Start-Process -Wait -FilePath ".\vc_redist.x64.exe" -ArgumentList "/install /quiet /norestart"
}
# --- Step 6: Run Tests ---
Write-Host "=== Step 6: Running all tests ===" -ForegroundColor Cyan
# Run UAP tests
Write-Host "--- Running UAP tests ---" -ForegroundColor Yellow
.\runtests.cmd * -HostingMode:UAP
# Run WPF tests
Write-Host "--- Running WPF tests ---" -ForegroundColor Yellow
.\runtests.cmd * -HostingMode:WPF
# Run Win32Explicit tests
Write-Host "--- Running Win32Explicit tests ---" -ForegroundColor Yellow
.\runtests.cmd * -HostingMode:Win32Explicit
# Run no-hosting-mode tests (unit tests, app tests)
Write-Host "--- Running unit/app tests ---" -ForegroundColor Yellow
.\runtests.cmd * -HostingMode:None
Pop-Location
Write-Host "=== All tests complete ===" -ForegroundColor Green
Notes
- ~8000 tests total, taking ~8 hours to run serially across all hosting modes.
- Tests require a visible desktop session. UI tests will fail if the Remote Desktop window is minimized, the screen is locked, or another window has focus. Look for errors like
RpcSetForegroundWindow or SetForegroundWindow: Failed.
- Managed tests (C#) can only run in WPF hosting mode (not UAP).
- Native tests (C++) can run in both UAP and WPF modes.
- Use
-TerminateOnFirstFailure for quick validation.
- Use
-Stat to see test statistics before running.
- Test output is in the
TestPayload\<platform><config>\Test\WexLogFileOutput\ folder.
Test Suite Reference
All test DLLs are in TestPayload\<platform><config>\Test\. Here is the complete listing:
External Tests (E2E UI integration — C++ native, require UAP or WPF hosting)
| DLL | Area |
|---|
External.Activation | App activation |
External.Adaptability | DPI/scaling/adaptability |
External.Automation | UI Automation / accessibility |
External.Controls | Built-in XAML controls (Button, ComboBox, CommandBar, etc.) |
External.Convergence | API convergence |
External.Enterprise | Enterprise scenarios (ListView, GridView, etc.) |
External.Foundation | Core rendering, images, text, media (includes AnimatedImageTests) |
External.Framework | XAML framework (binding, templates, resources) |
External.Quality | Visual quality tests |
External.Test | Test infrastructure validation |
External.Tools | App analysis tools |
External.Win32 | Win32/Desktop hosting |
Isolated Tests (Unit tests — C++ native, no hosting mode needed)
| DLL | Area |
|---|
Isolated.AccessKeys.* (6 DLLs) | Keyboard access keys |
Isolated.Adaptability.Qualifiers | Qualifier logic |
Isolated.Associative/Base/Collection/Colors/Com | Core data structures |
Isolated.Controls.Moco/Pivot/RelativePanel/Theming | Control-specific unit tests |
Isolated.Core.Gestures/Input | Touch/pointer input |
Isolated.Enterprise.* (2 DLLs) | ItemIndexRange, VisualStates |
Isolated.Foundation.Animation/Brushes/Elements/Math/Transforms | Foundation unit tests |
Isolated.Foundation.Imaging | Image decoding (includes AnimatedGifUnitTests) |
Isolated.Foundation.OfferableHeap/ThemeAnimationsHelper/Threading | Memory, animations, threading |
Isolated.Framework.* (9 DLLs) | Framework internals (CValue, Parser, DependencyObject, etc.) |
Isolated.Graphics/Legacy/Lifetime/Strings/Text | Various |
Isolated.Tools.AppAnalysis.*/XbfGenerator | Tooling |
Isolated.Xaml.AccessKeys.*/Focus.* | Focus/AccessKey subsystems |
Isolated.XamlDiagnostics | XAML diagnostics |
Managed Tests (C# — WPF hosting mode only)
| DLL | Area |
|---|
Managed.AccessKeys | Access keys (C#) |
Managed.Animation.EasingFunctionBaseTests | Animation easing |
Managed.Common | Common framework tests |
Managed.Controls | Controls (C#) |
Managed.Enterprise.Moco | Enterprise/Moco |
Managed.Foundation | Foundation (C#) |
Managed.Framework | Framework (C#) |
Managed.Media | Media |
Managed.Win32.Common/Hosting | Win32 hosting |
Other Test DLLs
| DLL | Area |
|---|
MUXControls.Test.dll | Modern WinUI controls (TreeView, NavigationView, InfoBar, etc.) |
UnitTests.dll | Core XAML unit tests |
DxamlCoreTipUnitTests.dll | Core tooltip unit tests |
Mapping: Source Code → Test DLL
| Source files changed in | Test DLL to run |
|---|
dxaml/xcp/** (core runtime) | External.Foundation, External.Framework, External.Controls |
controls/dev/** (WinUI controls) | MUXControls.Test.dll, External.Controls |
dxaml/xcp/components/imaging/** | Isolated.Foundation.Imaging |
dxaml/test/native/external/<area>/** | External.<Area> |
dxaml/test/resources/ (GIF/image assets) | Isolated.Foundation.Imaging, External.Foundation |
controls/test/** | MUXControls.Test.dll |
OSS Package Coherence
The OSS build of WinUI pins a small set of dependency packages directly in eng/Versions.props (the internal build ignores these and uses Maestro/Darc-managed versions instead). When those pins are not from the same internal build run, the symptoms are dramatic but the diagnosis is subtle.
Symptom recognition
If you see any of these, suspect package incoherence before suspecting your code:
- Mass
0xC0000602 (RoFailFastWithErrorContext) crashes on test startup — combase reports WinRT activation metadata mismatch.
- External category pass rate drops below ~80% (healthy is ~91%).
- Adaptability / Automation / Tools / Enterprise DLLs show cascading "Blocked" or crashes.
Microsoft.UI.Xaml.dll loads but a controls type fails to activate at runtime.- Same test passes in the internal CI but crashes locally on an OSS-restored build.
The IsInternalWinUIBuild gating pattern
eng/Versions.props (around line 21) defines:
<IsInternalWinUIBuild Condition="'$(IsInternalWinUIBuild)' == '' AND Exists('..\.azuredevops\')">true</IsInternalWinUIBuild>
The flag is true whenever ..\.azuredevops\ exists (i.e., when the repo is cloned from the ADO mirror, not from GitHub). Lines under Condition="'$(IsInternalWinUIBuild)' != 'true'" are the OSS-only pins — typically 5–7 packages:
Microsoft.Internal.FrameworkUdkMicrosoft.WindowsAppSDK.InteractiveExperiences (IXP) and its transport packageMicrosoft.WindowsAppSDK.Foundation and its transport packageMicrosoft.WindowsAppSDK.BaseMicrosoft.WindowsAppSDK.WinUIDetails
Internal builds read versions from eng/Version.Details.xml (Maestro dep-flow). OSS builds read the literal pins.
What "coherent" means
Two packages are coherent when their version strings prove they were produced by the same internal build run (byte-identical date stamp / build label, e.g. version suffixes ending in the same YYMMDD-HHMM.B token). Examples of incoherence that look fine to a human reviewer but break at runtime:
FrameworkUdk from build A combined with the FrameworkUdk DLL embedded inside the IXP package from build B — both packages claim the same nominal version, but the embedded DLL differs by KB.- MUX compiled against
WinUIDetails headers from build A, then linked at deployment time against the runtime IXP from build B.
Foundation and Base snapped from different days.
WinUIDetails is the secret killer
WinUIDetails is a headers-only NuGet package, but the headers ARE the compile-time ABI surface (vtable layouts, IIDs, struct sizes). If WinUIDetails is from a different build run than the runtime IXP / Foundation packages, MUX compiles successfully against one set of vtable layouts and then encounters a different set at activation time → 0xC0000602.
How to find a coherent snapshot
Two paths:
- Dep-flow / BAR (preferred) — query the BAR (Build Asset Registry) or use Darc to find a recent build that produced all the packages together. The build label suffix is the coherence proof.
- Cache inspection — check what's already on the machine:
packages\, packages_temp\, ~\.nuget\packages\ for matching date stamps across the OSS-pinned package set.
Validating the snapshot is published to the OSS feed
OSS builds restore from the WinUI-Dependencies feed on dev.azure.com/shine-oss. To verify a specific version is published before pinning:
$tok = (az account get-access-token --resource '499b84ac-1321-427f-aa17-267ca6975798' --query accessToken -o tsv)
$pkg = 'microsoft.internal.frameworkudk' # lowercase
Invoke-RestMethod -Uri "https://pkgs.dev.azure.com/shine-oss/microsoft-ui-xaml/_packaging/WinUI-Dependencies/nuget/v3-flatcontainer/$pkg/index.json" -Headers @{ Authorization = "Bearer $tok" }
A 200 with the version listed = ready to pin. A 404 = package needs to be promoted/published first.
The two-tier package consumption model
- Product DLL (
Microsoft.UI.Xaml.dll) consumes transport packages (*TransportPackage), gated by IsInternalWinUIBuild.
- Test/sample apps consume the flat packages (
LiftedIXPPackageName, FoundationPackageName) — and these versions are typically set unconditionally, not gated.
Both tiers must be coherent or runtime DLLs in TestPayload\ will conflict.
PR pattern when bumping OSS pins
A PR that updates the OSS-pinned versions:
- Touches only
eng/Versions.props (and eng/Version.Details.xml if needed for OSS-feed mirroring).
- All changes gated by
Condition="'$(IsInternalWinUIBuild)' != 'true'" so internal CI is unaffected.
- Must include a pass-rate validation table (see "PR + Validation Template" below).
- Acceptance bar: zero new failures vs unchanged main.
CI Test Invocation Map
When you need to know what CI runs and how it runs it, these two files are the source of truth:
-
build/AzurePipelinesTemplates/WinUI-CreateTestPayload-Job.yml — defines each test work item: testFilePathPattern (which DLL or AppX) + hostingMode.
-
Helix/GenerateHelixWorkItems.ps1 — look for the line that generates the TAEF command:
$taefExtraParameters = "/p:HostingMode=$HostingMode"
Every CI test invocation passes a HostingMode value to TAEF via /p:.
CI ↔ local mapping
| CI work item | Test pattern | Hosting mode | Local equivalent |
|---|
| MUXControlsInteractionTests | MUXControls.Test.dll | Default | runtests.cmd MUXControls.Test.dll |
| MUXControlsApiTests | UnpackagedApps\MUXControlsTestApp\MUXControlsTestApp.dll | Default | run from UnpackagedApps\MUXControlsTestApp\ |
| IXMPTests | IXMPTestApp.appx | Default | run from registered loose layout |
| MUXCoreTests | Microsoft.UI.Xaml.Tests.External.*.dll | UAP | runtests.cmd ... -HostingMode:UAP |
| MUXCoreTests-WPF | same | WPF | -HostingMode:WPF |
| MUXCoreManagedTests-WPF | Microsoft.UI.Xaml.Tests.Managed.*.dll | WPF | te.exe ... /p:HostingMode=WPF |
| MUXCoreTests-Win32Explicit | Microsoft.UI.Xaml.Tests.External.*.dll | Win32Explicit | -HostingMode:Win32Explicit |
| UnitTests (static / Isolated) | Microsoft.UI.Xaml.Tests.Isolated.*.dll | None | -HostingMode:None |
The Managed-tests gotcha (READ THIS BEFORE DEBUGGING "BLOCKED" MANAGED TESTS)
Managed tests are not AppX-hosted in CI. They run via te.processhost.exe with WPF XamlIsland hosting — i.e., they need /p:HostingMode=WPF.
A local runner that omits this flag will appear to "block all 700+ Managed tests" with 0x80070002 / 0x8007007E / 0x80000003 and lead the investigator down a .rd.xml / .NET Native rabbit hole that does not exist. The fix is one flag: /p:HostingMode=WPF.
This was a multi-week false wall. Don't repeat it.
ADO ↔ GitHub Mirror Reference
The WinUI3 source code lives in two clones, kept in sync by a one-way mirror:
- ADO
microsoft.visualstudio.com/WinUI/_git/microsoft-ui-xaml-lift (branch main) — source of truth for all mirrored content.
- GitHub
github.com/microsoft/microsoft-ui-xaml (branch winui3/main) — read-only mirror for mirrored files. A small set of GitHub-only files (typically OSS-only docs and samples not present in ADO) can and should be edited directly on GitHub.
The mirror pipeline is build/WinUI-MirrorMainSourceToExternalRepo-Official.yml.
Layout difference
The mirror uses targetRepositorySubdirectory: "src". A file at ADO eng/Versions.props lands at GitHub src/eng/Versions.props. The GitHub repo's main branch is a legacy OSS-only branch — do not edit it; the mirror writes to winui3/main.
Exclusions
build/PipelineScripts/WinUISourceMirroringExclusions.txt controls what does NOT mirror.
| Syntax | Meaning |
|---|
:/path/ | Exclude this path from the mirror |
:!/path/ | Re-include (negate) this path even if a parent is excluded |
:(glob,top)*.md | Top-level glob exclusion |
Useful when re-enabling a small subset of an excluded tree (e.g., excluding :/test/ but re-including :!/test/scripts/ for the local runner scripts).
"Which clone do I edit?" decision rule
- File exists in ADO under any mirrored path (most product/test source, this skill, etc.) → edit in ADO, create an ADO PR. Editing the GitHub copy will be silently overwritten on the next mirror sync.
- File exists only on GitHub (rare — typically OSS-only docs or samples added directly to the GitHub repo) → edit on GitHub.
- Submodules —
Samples/WinUIGallery is a submodule pointer to a separate repo. The submodule contents are not mirrored as plain files; only the pointer commit is.
Cross-Category Pass-Rate Reference
Expected pass-rate bands and approximate wall times for a healthy local validation run. Use these to calibrate "is my run healthy" before chasing failures.
| Category | Expected pass rate | Approx. test count | Approx. wall time | Notes |
|---|
Isolated (unit, hosting None) | ≥99% | ~720 | ~2 min | Fastest signal — run first |
| IXMPTestApp | 100% | 1 | <1 min | Smoke test for AppX deployment |
| MUXControlsTestApp | ~95% | ~600 | ~40 min | UI assertion drift expected |
| MUXControls.Test (sample of ~50) | ~92% | ~50 | ~20 min | Full suite ~825 tests, multi-hour |
| External (UAP + WPF + Win32Explicit) | ~91% | ~1,200 | ~3 h | Use 5s keepalive (see Reliability) |
Managed (HostingMode=WPF) | ~79–95% | ~720 | ~60–90 min | Requires /p:HostingMode=WPF |
| Combined OSS validation target | ≥94% | ~4,100 | ~4–5 h | A healthy OSS validation hits this band |
A pass rate that falls dramatically below the band for a category is almost always either:
- A package coherence issue (see OSS Package Coherence), or
- The runner being killed by idle-detection (see Reliability Tweaks).
It is rarely "your code change broke 30% of tests."
Troubleshooting & Known Issues
Build Errors
For build-related issues (PCH virtual memory exhaustion, stale precompiled headers, missing Spectre libs, Run a full init first, etc.), refer to the build skill which covers all build troubleshooting.
Test Payload Errors
CreateTestPayload.ps1 — robocopy exit codes
Symptom: Script appears to "fail" but robocopy exit codes below 8 are actually success.
Root Cause: Robocopy uses non-standard exit codes where 0-7 indicate various forms of success. The script handles this internally.
Fix: Only exit codes 8+ are actual failures. Check CreateTestPayload.log in the TestPayload directory for details.
TestPayload is empty or missing files
Symptom: Test payload directory exists but is missing test DLLs or TAEF binaries.
Root Cause: Build didn't produce all outputs (partial build, or wrong build target).
Fix: Ensure you ran a full build (not just product or mux):
.\initrun.ps1 .\build.cmd /q /b # builds product + tests
Then recreate the payload:
.\initrun.ps1 powershell -ExecutionPolicy Bypass -File test\CreateTestPayload.ps1 -Platform x64 -Configuration chk -Clean
Machine Setup Errors
testmachine-prerun.cmd fails with access denied
Symptom: Registry operations or AppX installs fail with permission errors.
Fix: Must run from an elevated (admin) PowerShell or cmd prompt.
AppX package install error 0x80073D06 or 0x80073CFB
Symptom: Add-AppxPackage reports an error during machine setup.
Root Cause: A same or newer version of the package is already installed.
Fix: Safe to ignore — the setup script already handles this and prints "The same or higher version of this package is already installed."
Test Execution Errors
SetForegroundWindow: Failed / RpcSetForegroundWindow errors
Symptom: Tests fail because the test app window cannot be brought to the foreground.
Root Cause: UI tests require an active, visible desktop session. The session is either minimized, locked, or another window has focus.
Fix:
- If using Hyper-V: Use basic "Virtual Machine Connection" (vmconnect.exe), NOT Enhanced Session
- If using Remote Desktop: Keep the RDP window visible and focused, do not minimize it
- Do not lock the screen while tests are running
- Check View menu in VM connect to ensure you're NOT using Enhanced Session
Tests show "blocked" status instead of running
Symptom: Managed C# tests report "blocked" and don't execute.
Root Cause: Managed (.NET) tests cannot run in UAP hosting mode. They must use WPF mode.
Fix: Run managed tests with WPF hosting:
.\runtests.cmd *ManagedTestName* -HostingMode:WPF
COM_END WindowHelper::VerifyTestCleanup: Error: Caught Platform::Exception^: Unspecified error
Symptom: Tests fail with COM/platform errors during cleanup.
Root Cause: Usually caused by running tests over Remote Desktop or Hyper-V Enhanced Session.
Fix: Switch to basic VM connection (not Enhanced Session, not RDP). See the SetForegroundWindow fix above.
Tests fail to find test DLLs
Symptom: te.exe reports it cannot find test binaries.
Root Cause: Running runtests.cmd from the wrong directory.
Fix: You must cd into the TestPayload directory first:
cd <repo-root>\TestPayload\x64chk
.\runtests.cmd *MyTest*
Query matched tests requiring conflicting HostingModes
Symptom: runtests.cmd auto-detection fails because the query matches tests in multiple hosting modes.
Root Cause: A broad wildcard (e.g., *) matches tests that require different hosting modes, and the script can only run one mode at a time.
Fix: Explicitly specify the hosting mode:
.\runtests.cmd * -HostingMode:UAP
.\runtests.cmd * -HostingMode:WPF
.\runtests.cmd * -HostingMode:Win32Explicit
.\runtests.cmd * -HostingMode:None
Init / Build Errors
For init and build-related issues (PCH virtual memory exhaustion, stale precompiled headers, missing Spectre libs, NuGet authentication, dotnet-install failures, etc.), refer to the build skill (/.github/skills/build/SKILL.md) which covers all build and init troubleshooting.
HRESULT Decoder Table
Every crash code we've encountered, what it actually means, and the first thing to try.
| HRESULT | Symbolic name | Common cause | First fix |
|---|
0x80073CF6 / 0x80073CFB / 0x80073D06 | APPX_E_* install conflicts | Same or newer version of an AppX is already registered | Benign in dev mode — loose layout still works. Uninstall the conflicting registration only if it actively prevents activation. |
0x80070002 | ERROR_FILE_NOT_FOUND | TAEF DeploymentItem source missing in deployment root | Stage the missing file from BuildOutput\bin\<flavor>\Test\ to the AppX deployment root (parent of the AppX folder). |
0x8007007E | ERROR_MOD_NOT_FOUND | Transitive DLL dependency missing at load time | Bulk-copy from Test\UnpackagedApps\MUXControlsTestApp\ (or the equivalent staging dir) into the deployment root. |
0x80080204 | AppX activation failure | Manifest/identity mismatch, or TAEF launching the wrong host EXE | Verify the registered AppX matches what TAEF is launching; confirm the AppX manifest entry point. |
0x80000003 | Wex.Common.dll DebugBreak | Test framework hit a native breakpoint — usually because a test DLL is missing from inside the registered AppX | Re-stage test DLLs into the AppX install location (not just the deployment root parent). |
0x8027025B | AppX activation failed | IApplicationActivationManager couldn't activate the identity | Confirm AppX is registered AND the test cert is in TrustedPeople. |
0xC0000602 | RoFailFastWithErrorContext | WinRT activation metadata mismatch — almost always package incoherence | See OSS Package Coherence section above. Don't debug your code first. |
0x80004005 | E_FAIL from TAEF | TAEF received a target file type it doesn't accept (e.g. .msix instead of , or a DLL outside its registered AppX install location) |
Baseline Failure Catalog
Before treating any failure as "caused by my change", reproduce it on unchanged main with the same package pins and the same TestPayload state. If it reproduces, it's pre-existing baseline drift, not your bug.
Categories of baseline failures we've consistently observed
- External (component) DLLs — typically 80–110 pre-existing failures on a healthy ~91% pass rate. Distribution roughly:
Test, Win32, Convergence — 3–4 failures each.Adaptability — ~6 failures.Enterprise — ~58–74 failures (largest contributor).Tools — ~4–11 failures.- These persist across package version changes and across many weeks.
- MUXControlsTestApp (~95% expected) — UI-assertion drift is expected. Examples we've reproduced on unchanged main:
RatingControlTests tap-and-return offsets, asynchronous-timing assertions in animation tests.
- Isolated — 1–2 single flakes (often clean on retry). Expected ≥99% pass.
- Managed (
HostingMode=WPF) — some Win32.Common deterministic failures show up in the ~79% pass-rate sample.
What is NOT a baseline failure
- Any cascading
0xC0000602 activation crash — that's package incoherence.
- A sudden surge of "Blocked" results across multiple DLLs — that's the runner being killed by idle-detection (see Reliability Tweaks), not a real failure.
- A whole category dropping below its expected band — investigate the package set, not the individual tests.
Test Runner Reliability Tweaks
Patterns that prevent the most common run-time waste during long local validation runs.
5-second keepalive for long External runs
Without keepalive, idle-detection kills External DLL runs after a period of no input — and produces hundreds of "Blocked" results instead of real pass/fail. Two options:
- Pass a TAEF foreground / idle-keepalive option appropriate for your TAEF version.
- Run a parallel mouse-wiggle script: move the cursor by 1 pixel every ~5–15 seconds. This keeps
SetForegroundWindow permission alive on the active session, which the UI tests need.
Background-shell pattern for long-running DLLs
Anything that runs for ≥5 minutes (Controls = 7+ min, Enterprise / Tools / Framework, full External run) should be launched detached so it survives Copilot CLI / shell restart:
Start-Process pwsh -WindowStyle Normal -ArgumentList '-NoExit','-Command',"<run command> *> '<per-DLL log>'; Add-Content '<csv>' '<dll>,<status>'"
Tee output to a per-DLL log file and append a one-line CSV summary at completion — that gives you an audit trail across the full multi-hour run.
Crash dumps via WER LocalDumps
Configure HKCU once so any test-host crash produces a dump:
HKCU\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\te.processhost.exe
HKCU\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\TE.exe
DumpFolder = <writable path>
DumpType = 2 (full dump)
Critical when chasing combase fast-fail crashes.
TAEF /select filter is unreliable
TE.exe <dll> /select:"@Name='X' OR @Name='Y'" sometimes runs the full suite anyway. When sampling for diagnostic signal:
- Enumerate first:
TE.exe <dll> /list → capture the full test name list.
- Run the first N test names individually, not via
/select.
A 10-tests-per-DLL sample across all DLLs in a category gives ~80% of the diagnostic signal in 10–15% of the wall time. Sample by running fewer DLLs, not by trying to filter inside a DLL.
TestPayload directory naming
Build outputs land at BuildOutput\bin\amd64chk\ but TestPayload lands at TestPayload\x64chk\ — same architecture, different naming convention. This is normal; don't try to "fix" it.
AppX-hosted tests need the right deployment root
For AppX-hosted tests, target the test DLL inside the registered loose layout, not under the raw TestPayload\ directory. The deployment root for DeploymentItem resolution is the parent of the AppX folder.
PR + Validation Template
For changes that affect the OSS test pipeline (Versions.props pins, mirror exclusions, test infra scripts):
Always include a pass-rate validation table
Category Total Run Pass Fail Pass% Time
Isolated (unit) 718 718 717 1 99.86% ~2m
IXMPTestApp 1 1 1 0 100.00% ~30s
MUXControls.Test (sample) 825 54 50 4 92.59% ~19m
MUXControlsTestApp 601 543 517 26 95.21% ~38m
External (component) 1,225 1,225 1,121 104 91.51% ~3h
Managed (.NET, WPF host) 724 ... ... ... ~79% ~60-90m
TOTAL 4,094 ... ... ... ≥94% ~4.5h
Failure analysis paragraph
State explicitly whether failures reproduce on unchanged main:
- Reproduces on unchanged main → pre-existing baseline drift, not introduced by this PR.
- Does not reproduce on unchanged main → real regression, must be investigated before merge.
The acceptance bar is "PR introduces zero new failures."
Gating note
When the change is gated by Condition="'$(IsInternalWinUIBuild)' != 'true'", say so explicitly so reviewers know the internal CI build is unaffected and only the GitHub OSS build picks up the change.
ADO PR description size
ADO PR descriptions have a 4000-character hard limit. If your validation report is larger:
- Truncate aggressively in the PR body.
- Keep the validation table.
- Drop per-test failure breakdowns; link instead.
Which repo to PR to
Per the ADO ↔ GitHub Mirror Reference above:
- File exists in ADO under any mirrored path → ADO PR.
- File exists only on GitHub → GitHub PR.
- When in doubt, ADO PR — the mirror will propagate it.
