run-tests

> Source: vendored from [dotnet/skills](https://github.com/dotnet/skills/blob/main/plugins/dotnet-test/skills/run-tests/SKILL.md). # Run .NET Tests Detect the test platform and framework, run tests, and apply filters using `dotnet test`. ## When to Use - User wants to run tests in a .NET project - User needs to run a subset of tests using filters - User needs help detecting which test platform (VSTest vs MTP) or framework is in use - User wants to understand the correct filter syntax for their setup ## When Not to Use - User needs to write or generate test code (use `writing-mstest-tests` for MSTest, or general coding assistance for other frameworks) - User needs to migrate from VSTest to MTP (use `migrate-vstest-to-mtp`) - User wants to iterate on failing tests without rebuilding (use `mtp-hot-reload`) - User needs CI/CD pipeline configuration (use CI-specific skills) - User needs to debug a test (use debugging skills) ## Inputs | Input | Required | Description | |-------|----------|-------------| | Project or solution path | No | Path to the test project (.csproj) or solution (.sln, .slnf, .slnx). Defaults to current directory. | | Filter expression | No | Filter expression to select specific tests | | Target framework | No | Target framework moniker to run against (e.g., `net8.0`) | ## Critical Rules — Avoid Cross-Platform Mistakes These are the most common agent mistakes. Internalize before proceeding: | Rule | Why | |------|-----| | **Do NOT use `--logger trx`** for MTP projects | MTP uses `--report-trx` (requires the TrxReport extension package) | | **Do NOT use `--report-trx`** for VSTest projects | VSTest uses `--logger trx` | | **Do NOT use `-- --arg`** on .NET SDK 10+ | SDK 10+ passes MTP args directly: `dotnet test --project . --report-trx` | | **Do NOT omit `--`** on .NET SDK 8/9 with MTP | SDK 8/9 requires the separator: `dotnet test -- --report-trx` | | **Do NOT use `--filter "ClassName=..."`** with xUnit v3 on MTP | xUnit v3 on MTP uses `--filter-class`, `--filter-method`, `--filter-trait` | | **Do NOT use bare positional path** on SDK 10+ | Use `--project <path>` or `--solution <path>` instead | | **Do NOT use `--blame`** for MTP projects | MTP uses `--blame-crash` and `--blame-hang-timeout` separately (each requires its extension package) | | **Do NOT use `--collect "Code Coverage"`** for MTP | MTP uses `--coverage` (requires the CodeCoverage extension package) | ## Workflow ### Quick Reference | Platform | SDK | Command pattern | |----------|-----|----------------| | VSTest | Any | `dotnet test [<path>] [--filter <expr>] [--logger trx]` | | MTP | 8 or 9 | `dotnet test [<path>] -- <MTP_ARGS>` | | MTP | 10+ | `dotnet test --project <path> <MTP_ARGS>` | **Detection files to always check** (in order): `global.json` -> `.csproj` -> `Directory.Build.props` -> `Directory.Packages.props` **If the prompt names a subset of tests** (e.g., "integration tests", "smoke tests", a specific class, a specific TFM), plan to apply the matching filter / `--framework` in [Step 3](#step-3-run-filtered-tests) — do not run the whole suite. ### Step 1: Detect the test platform and framework 1. Run `dotnet --version` in the project directory to determine the SDK version. This accounts for `global.json` SDK pinning. 2. Read `global.json` — on .NET SDK 10+, `"test": { "runner": "Microsoft.Testing.Platform" }` is the **authoritative MTP signal**. If present, the project uses MTP and SDK 10+ syntax (no `--` separator). 3. Read `.csproj`, `Directory.Build.props`, **and** `Directory.Packages.props` for framework packages and MTP properties. **Always check all three files** — MTP properties are frequently set in `Directory.Build.props` rather than individual `.csproj` files. 4. For full detection logic (SDK 8/9 signals, framework identification), see the `platform-detection` skill. **What to look for in each file:** | File | Look for | Indicates | |------|----------|-----------| | `global.json` | `"test": { "runner": "Microsoft.Testing.Platform" }` | MTP on SDK 10+ | | `global.json` | `"sdk": { "version": "..." }` | SDK version (determines `--` separator behavior) | | `.csproj` | `<TestingPlatformDotnetTestSupport>true` | MTP on SDK 8/9 | | `.csproj` | `MSTest`, `xunit.v3`, `NUnit`, `TUnit` packages | Framework identity | | `.csproj` | `Microsoft.NET.Test.Sdk` + test adapter | VSTest (unless overridden by MTP signals above) | | `.csproj` | `<TargetFrameworks>` (plural) | Multi-TFM — may need `--framework` | | `Directory.Build.props` | `<TestingPlatformDotnetTestSupport>true` | MTP on SDK 8/9 (often set here, not in .csproj) | | `Directory.Packages.props` | Centrally managed test package versions | Framework identity for CPM repos | **Quick detection summary:** | Signal | Means | |--------|-------| | `global.json` has `"test": { "runner": "Microsoft.Testing.Platform" }` | **MTP on SDK 10+** — pass args directly, no `--` | | `<TestingPlatformDotnetTestSupport>true` in csproj or Directory.Build.props | **MTP on SDK 8/9** — pass args after `--` | | Neither signal present | **VSTest** | ### Step 2: Run tests #### VSTest (any .NET SDK version) ```bash dotnet test [<PROJECT> | <SOLUTION> | <DIRECTORY> | <DLL> | <EXE>] ``` Common flags: | Flag | Description | |------|-------------| | `--framework <TFM>` | Target a specific framework in multi-TFM projects (e.g., `net8.0`) | | `--no-build` | Skip build, use previously built output | | `--filter <EXPRESSION>` | Run selected tests (see [Step 3](#step-3-run-filtered-tests)) | | `--logger trx` | Generate TRX results file | | `--collect "Code Coverage"` | Collect code coverage using Microsoft Code Coverage (built-in, always available) | | `--blame` | Enable blame mode to detect tests that crash the host | | `--blame-crash` | Collect a crash dump when the test host crashes | | `--blame-hang-timeout <duration>` | Abort test if it hangs longer than duration (e.g., `5min`) | | `-v <level>` | Verbosity: `quiet`, `minimal`, `normal`, `detailed`, `diagnostic` | #### MTP with .NET SDK 8 or 9 With `<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>`, `dotnet test` bridges to MTP but uses VSTest-style argument parsing. MTP-specific arguments must be passed after `--`: ```bash dotnet test [<PROJECT> | <SOLUTION> | <DIRECTORY> | <DLL> | <EXE>] -- <MTP_ARGUMENTS> ``` #### MTP with .NET SDK 10+ With the `global.json` runner set to `Microsoft.Testing.Platform`, `dotnet test` natively understands MTP arguments without `--`: ```bash dotnet test [--project <PROJECT_OR_DIRECTORY>] [--solution <SOLUTION_OR_DIRECTORY>] [--test-modules <EXPRESSION>] [<MTP_ARGUMENTS>] ``` Examples: ```bash # Run all tests in a project dotnet test --project path/to/MyTests.csproj # Run all tests in a directory containing a project dotnet test --project path/to/ # Run all tests in a solution (sln, slnf, slnx) dotnet test --solution path/to/MySolution.sln dotnet test --solution path/to/MySolution.slnf dotnet test --solution path/to/MySolution.slnx # Run all tests in a directory containing a solution dotnet test --solution path/to/ # Run with MTP flags dotnet test --project path/to/MyTests.csproj --report-trx --blame-hang-timeout 5min ``` > **Note**: The .NET 10+ `dotnet test` syntax does **not** accept a bare positional argument like the VSTest syntax. Use `--project`, `--solution`, or `--test-modules` to specify the target. #### Common MTP flags These flags apply to MTP on both SDK versions. On SDK 8/9, pass after `--`; on SDK 10+, pass directly. > **Important:** `dotnet test`/MSBuild flags such as `--framework`, `--no-build`, `--configuration`, and `--verbosity` are consumed by `dotnet test` itself (they drive restore/build/host selection) and **always go BEFORE `--`**, regardless of platform or SDK. Only MTP test-platform arguments go after `--` on SDK 8/9. For example: `dotnet test --framework net9.0 -- --report-trx` (built-in flag before `--`, MTP extension flag after). **Built-in flags (always available):** | Flag | Description | |------|-------------| | `--results-directory <DIR>` | Directory for test result output | | `--diagnostic` | Enable diagnostic logging for the test platform | | `--diagnostic-output-directory <DIR>` | Directory for diagnostic log output | **Extension-dependent flags (require the corresponding extension package to be registered):** | Flag | Requires | Description | |------|----------|-------------| | `--filter <EXPRESSION>` | Framework-specific (not all frameworks support this) | Run selected tests (see [Step 3](#step-3-run-filtered-tests)) | | `--report-trx` | `Microsoft.Testing.Extensions.TrxReport` | Generate TRX results file | | `--report-trx-filename <FILE>` | `Microsoft.Testing.Extensions.TrxReport` | Set TRX output filename | | `--blame-hang-timeout <duration>` | `Microsoft.Testing.Extensions.HangDump` | Abort test if it hangs longer than duration (e.g., `5min`) | | `--blame-crash` | `Microsoft.Testing.Extensions.CrashDump` | Collect a crash dump when the test host crashes | | `--coverage` | `Microsoft.Testing.Extensions.CodeCoverage` | Collect code coverage using Microsoft Code Coverage | > Some frameworks (e.g., MSTest) bundle common extensions by default. Others may require explicit package references. If a flag is not recognized, check that the corresponding extension package is referenced in the project. #### Alternative MTP invocations MTP test projects are standalone executables. Beyond `dotnet test`, they can be run directly: ```bash # Build and run dotnet run --project <PROJECT_PATH> # Run a previously built DLL dotnet exec <PATH_TO_DLL> # Run the executable directly (Windows) <PATH_TO_EXE> ``` These alternative invocations accept MTP command line arguments directly (no `--` separator needed). ### Step 3: Run filtered tests See the `filter-syntax` skill for the complete filter syntax for each platform and framework combination. Key points: - **VSTest** (MSTest, xUnit v2, NUnit): `dotnet test --filter <EXPRESSION>` with `=`, `!=`, `~`, `!~` operators - **MTP -- MSTest and NUnit**: Same `--filter` syntax as VSTest; pass after `--` on SDK 8/9, directly on SDK 10+ - **MTP -- xUnit v3**: Uses `--filter-class`, `--filter-method`, `--filter-trait` (not VSTest expression syntax). For a **single combined expression** (e.g., a class-name pattern AND a trait), use `--filter-query` with the xUnit v3 query filter language: path segments `/<assembly>/<namespace>/<class>/<method>` with `*` wildcards and a `[Trait=Value]` qualifier — for example `dotnet test -- --filter-query "/*/*/*IntegrationTests*/*[Category=Smoke]"`. See the `filter-syntax` skill for the full query language. - **MTP -- TUnit**: Uses `--treenode-filter` with path-based syntax #### When the user names a test category, trait, or group When the prompt names a subset of tests by category (e.g., "integration tests", "unit tests", "smoke tests", "fast tests"), **do not run all tests** — translate the user's vocabulary into the platform-appropriate filter: 1. **Inspect the test source files** for filter-attribute annotations that match the named group: | Framework | Attribute | Filter property |