csharp-spector-coverage-gaps

# Discovering and implementing Spector coverage gaps for http-client-csharp ## Overview This skill discovers which Spector scenarios the C# emitter (`@typespec/http-client-csharp`) does **not** yet cover, then implements the missing test(s). Spector scenarios are defined in `@typespec/http-specs` and `@azure-tools/azure-http-specs`. The coverage dashboard is at <https://typespec.io/can-i-use/http/>. > **Note:** `{PKG}` refers to `<repo-root>/packages/http-client-csharp` throughout this document. ## Inputs You may receive one of: - **"Find coverage gaps"** — discover all missing scenarios and present them. - **Spector spec link** — a link to a specific spec under `packages/http-specs/specs/...` or `packages/azure-http-specs/specs/...`. - **Spec name** — e.g., `http/encode/duration`, `http/type/model/flatten`, `http/type/union/discriminated`. ## Output - A report of coverage gaps (when discovering). - New or updated C# NUnit test file(s) under `{PKG}/generator/TestProjects/Spector.Tests/Http/`. - Updated `{PKG}/generator/TestProjects/Spector.Tests/TestProjects.Spector.Tests.csproj` if a new project reference is needed. - Regenerated client code (via `Generate.ps1`). ## Workflow - [ ] Ensure prerequisites are met (npm ci, npm run build) - [ ] Discover coverage gaps or identify the target spec - [ ] Verify the spec is not in the failing specs list - [ ] Generate the C# client for the target spec (unstubbed with `-Stubbed $false`, or use Test-Spector.ps1) - [ ] Read the generated client to understand the API surface - [ ] Read the TypeSpec scenario file to understand expected behavior - [ ] Find or create the test file - [ ] Implement the test(s) following existing conventions - [ ] Run the tests using `Test-Spector.ps1 -filter "<spec-path>"` - [ ] Validate all tests pass --- ## Prerequisites — Environment setup Before starting, ensure the build environment is ready. These instructions **supersede** the repo-root `pnpm` instructions. 1. **Install dependencies** (from `{PKG}`): ```powershell cd {PKG} npm ci ``` 2. **Build the package**: ```powershell npm run build ``` > ⚠️ Do NOT run `pnpm install` or `pnpm build` at the repo root — only the http-client-csharp package build is needed. --- ## Step 1 — Discover coverage gaps ### How specs are filtered The file `{PKG}/eng/scripts/Spector-Helper.psm1` defines which specs are included/excluded: **Failing specs** are defined in `{PKG}/eng/scripts/Spector-Helper.psm1` in the `$failingSpecs` array. Always check that file for the current list — do not hardcode specs here. **Azure allow-list** (only these Azure specs are tested): - `http/client/structure/client-operation-group` - `http/client/structure/default` - `http/client/structure/multi-client` - `http/client/structure/renamed-operation` - `http/client/structure/two-operation-group` - `http/resiliency/srv-driven` All other specs from `@typespec/http-specs` are included by default. ### Discovering gaps programmatically Compare available specs against existing tests. Run this from `{PKG}`: ```powershell Import-Module "{PKG}/eng/scripts/Spector-Helper.psm1" -DisableNameChecking -Force # Get all valid specs $specs = Get-Sorted-Specs | ForEach-Object { Get-SubPath $_ } # Get all test files $testFiles = Get-ChildItem -Path "{PKG}/generator/TestProjects/Spector.Tests/Http" -Recurse -Filter "*Tests.cs" | ForEach-Object { $_.FullName } # For each spec, check if a corresponding test directory/file exists foreach ($spec in $specs) { $specParts = $spec -replace '/', '\' -split '\\' # Convert to expected test path segments $testPath = "{PKG}/generator/TestProjects/Spector.Tests/Http" # ... check if test exists } ``` Alternatively, **manually compare** the list of specs against existing test files: **Existing test coverage** (test directories under `Spector.Tests/Http/`): ``` Authentication/ApiKey, Authentication/Http/Custom, Authentication/OAuth2, Authentication/Union Client/Naming, Client/Structure/{ClientOperationGroup,Default,MultiClient,RenamedOperation,TwoOperationGroup} Documentation Encode/{Array,Bytes,DateTime,Numeric} Parameters/{Basic,BodyOptionality,CollectionFormat,Path,Query,Spread} Payload/{ContentNegotiation,JsonMergePatch,MediaType,Multipart,Pageable,Xml} Resiliency/SrvDriven/{V1,V2} Response/StatusCodeRange Routes Serialization/EncodedName/Json Server/{Endpoint/NotDefined,Path/Multiple,Path/Single,Versions/NotVersioned,Versions/Versioned} SpecialHeaders/{ConditionalRequest,Repeatability} Versioning/{Added,MadeOptional,Removed,RenamedFrom,ReturnTypeChangedFrom,TypeChangedFrom} _Type/{Dictionary,Model/Empty,Model/Inheritance/*,Model/Usage,Model/Visibility,Property/*,Scalar,Union,_Array,_Enum/*} ``` ### Accurate gap detection The naive approach of matching spec paths to test directory names can produce false positives because: - C# reserved words get underscore-prefixed in test dirs (e.g., `type/` → `_Type/`, `array` → `_Array`, `enum` → `_Enum`) - kebab-case gets converted to PascalCase (e.g., `content-negotiation` → `ContentNegotiation`) - Some test files cover a parent spec but not sub-specs (e.g., `UnionTests.cs` covers `type/union` but NOT `type/union/discriminated`) **To find real gaps**, verify each candidate by checking whether a test _directory_ exists for the spec's exact path, including sub-paths. A test file at a parent level does NOT cover child specs. **Known failing specs** are tracked in `$failingSpecs` in `{PKG}/eng/scripts/Spector-Helper.psm1`. Always read that file for the current list. > **Important:** The gap list evolves over time. Always re-run the comparison to get current gaps. All committed Spector libraries are stubbed — `[SpectorTest]` will auto-skip tests unless you regenerate with `-Stubbed $false` or use `Test-Spector.ps1`. ### Understanding stubbed vs unstubbed generation All Spector libraries **committed to the repository are stubbed**. `Generate.ps1` defaults to `$Stubbed = $true`, which passes `--option @typespec/http-client-csharp.generator-name=StubLibraryGenerator` to the emitter. Stubbed clients use expression-bodied constructors (`=>`) instead of block bodies (`{ }`), and the `[SpectorTest]` attribute automatically skips tests for stubbed clients. To generate **unstubbed** code (for local testing), pass `-Stubbed $false`: ```powershell pwsh eng/scripts/Generate.ps1 -filter "<spec-path>" -Stubbed $false ``` However, the recommended way to test is via `Test-Spector.ps1` (see Step 7), which handles the unstubbed regeneration, test execution, and directory restoration automatically. --- ## Step 2 — Read the spec to understand expected behavior Spec files live in `{PKG}/node_modules/@typespec/http-specs/specs/` (or `@azure-tools/azure-http-specs/specs/` for Azure specs). If node_modules is not installed, read from the source at `<repo-root>/packages/http-specs/specs/`. Each spec contains: - **`main.tsp`** — the TypeSpec definition with `@scenario` and `@scenarioDoc` decorators - **`client.tsp`** (optional) — client-level customizations; takes priority over `main.tsp` during generation - **`tspconfig.yaml`** (optional in the Spector test project) — C#-specific generation options Read the `@scenarioDoc` decorators to understand: - The HTTP method, path, and expected parameters - The expected request body shape - The expected response status code and body - Any special behavior (e.g., "should return 204", "should send header X") --- ## Step 3 — Generate the C# client for the target spec Use `Generate.ps1` with a filter to generate only the specific spec. By default, `Generate.ps1` generates **stubbed** code (same as what is committed to the repo). To generate **unstubbed** code for local testing, pass `-Stubbed $false`: ```powershell cd {PKG} # Generate unstubbed (for local testing/development) pwsh eng/scripts/Generate.ps1 -filter "<spec-path>" -Stubbed $false # Generate stubbed (default, matches what is committed to repo) pwsh eng/scripts/Generate.ps1 -filter "<spec-path>" ``` Examples: ```powershell # Single spec (unstubbed for testing) pwsh eng/scripts/Generate.ps1 -filter "http/encode/duration" -Stubbed $false # Versioning spec (generates v1 + v2 automatically) pwsh eng/scripts/Generate.ps1 -filter "http/versioning/added" -Stubbed $false ``` The generated code lands in `{PKG}/generator/TestProjects/Spector/<spec-path>/src/Generated/`. ### Verify generation succeeded ```powershell # Check that client code was generated Get-ChildItem "{PKG}/generator/TestProjects/Spector/<spec-path>/src/Generated/" -Filter "*Client.cs" ``` > **Note:** All committed Spector libraries are stubbed. The `[SpectorTest]` attribute automatically skips tests for stubbed clients. Use `Test-Spector.ps1` (Step 7) to regenerate unstubbed, run tests, and restore automatically. --- ## Step 4 — Read the generated client API surface Browse the generated code to understand: 1. **Client class(es)**: `*Client.cs` — the entry point(s) 2. **Sub-clients**: accessed via `Get*Client()` methods 3. **Operations**: async methods like `GetAsync()`, `PutAsync(body)`, `SendAsync()` 4. **Models**: under `Models/` — request/response shapes 5. **Constructor signature**: `new XClient(Uri endpoint, XClientOptions options)` or `new XClient(Uri endpoint, KeyCredential credential, XClientOptions options)` Pay attention to: - Method names (they map to TypeSpec operation names) - Parameter types (models, primitives, BinaryData) - Return types (`ClientResult`, `ClientResult<T>`, `AsyncPageable<T>`) --- ## Step 5 — Create or extend the test file ### Directory and namespace conventions The test directory structure mirrors the spec path with these transformations: - `http/` → `Http/` - kebab-case → PascalCase (e.g., `content-negotiation` → `ContentNegotiation`) - `type/` → `_Type/` (leading underscore because `Type` is a C# keyword) - `array` → `_Array` (same reason) - `enum` → `_Enum` (same reason) **Namespace pattern**: `TestProjects.Spector.Tests.Http.<PascalCasePath>` Example mappings: | Spec path | Test directory | Namespace | | ------------------------------- | --------------------------------- | ----------------------------------------------------------- | | `http/encode/duration` | `Http/Encode/Duration/` | `TestProjects.Spector.Tests.Http.Encode.Duration` | | `http/type/union/discriminated` | `Http/_Type/Union/Discriminated/` | `TestProjects.Spector.Tests.Http._Type.Union.Discriminated` | | `http/special-words` | `Http/SpecialWords/` | `TestProjects.Spector.Tests.Http.SpecialWords` | ### Test file template ```csharp // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. using System.Threading.Tasks; using <GeneratedNamespace>; using NUnit.Framework; namespace TestProjects.Spector.Tests.Http.<Category>.<SubCategory> { public class <Name>Tests : SpectorTestBase { [SpectorTest] public Task <ScenarioName>() => Test(async (host) => { var response = await new <Client>(host, null).<Method>Async(<args>); Assert.AreEqual(<expectedStatusCode>, response.GetRawResponse().Status); }); } } ``` ### Common test patterns **Simple void operation (204 response):** ```csharp [SpectorTest] public Task SimpleOp() => Test(async (host) => { var response = await new MyClient(host, null).DoThingAsync(); Assert.AreEqual(204, response.GetRawResponse().Status); }); ``` **GET with typed response:** ```csharp [SpectorTest] public Task GetValue() => Test(async (host) => { var response = await new MyClient(host, null).GetValueAsync(); Assert.AreEqual("expected", response.Value); }); ``` **GET with model response:** ```csharp [SpectorTest] public Task GetModel() => Test(async (host) => { var response = await new MyClient(host, null).GetModelAsync(); Assert.AreEqual("name", response.Value.Name); Assert.AreEqual(42, response.Value.Age); }); ``` **PUT/POST with body:** ```csharp [SpectorTest] public Task SendModel() => Test(async (host) => {