csharp-azure-spector-coverage-gaps

# Discovering and implementing Spector coverage gaps for the Azure C# emitter ## Overview This skill discovers which Spector scenarios the Azure C# emitter (`@azure-tools/typespec-csharp`) does **not** yet cover, then implements the missing test(s). Spector scenarios are defined in two packages: - **Standard specs**: `@typespec/http-specs` — general HTTP client scenarios - **Azure specs**: `@azure-tools/azure-http-specs` — Azure-specific scenarios (core, LRO, paging, resource-manager, etc.) Two coverage dashboards exist: | Dashboard | URL | Scope | |-----------|-----|-------| | **Standard** | <https://typespec.io/can-i-use/http/> | Standard HTTP specs only | | **Azure** | <https://azure.github.io/typespec-azure/can-i-use/http/> | Standard + Azure specs | > **Tip:** Comparing the two dashboards reveals Standard scenarios where the Azure emitter lags behind (or ahead of) the core TypeSpec C# emitter in `microsoft/typespec`. > **Note:** `{PKG}` refers to `<repo-root>/eng/packages/http-client-csharp` throughout this document. ## Inputs You may receive one of: - **"Find coverage gaps"** — discover all missing scenarios and present them. - **"Compare Azure vs Standard dashboards"** — identify Standard scenarios covered on typespec.io but not on the Azure dashboard (or vice versa). - **Spector spec link** — a link to a specific spec under `packages/http-specs/specs/...` or `packages/azure-http-specs/specs/...`. - **Spec name** — e.g., `http/encode/duration`, `http/azure/core/lro/standard`, `http/type/union/discriminated`. ## Output - A report of coverage gaps (when discovering). - A comparison table of Standard vs Azure dashboard coverage (when comparing). - New or updated C# NUnit test file(s) under `{PKG}/generator/TestProjects/Spector.Tests/Http/`. - Updated `{PKG}/generator/TestProjects/Spector.Tests/TestProjects.Spector.Tests.csproj` if a new project reference is needed. - Regenerated client code (via `Generate.ps1`). ## Workflow - [ ] Ensure prerequisites are met (npm ci, npm run build) - [ ] Discover coverage gaps or identify the target spec - [ ] Verify the spec is not in the failing specs list - [ ] Generate the C# client for the target spec (unstubbed, or use Test-Spector.ps1) - [ ] Read the generated client to understand the API surface - [ ] Read the TypeSpec scenario file to understand expected behavior - [ ] Find or create the test file - [ ] Implement the test(s) following existing conventions - [ ] Run the tests using `Test-Spector.ps1 -filter "<spec-path>"` - [ ] Validate all tests pass --- ## Prerequisites — Environment setup Before starting, ensure the build environment is ready. 1. **Install dependencies** (from `{PKG}`): ```powershell cd {PKG} npm ci ``` 2. **Build the package**: ```powershell npm run build ``` > ⚠️ Do NOT run `pnpm install` or `pnpm build` at the repo root — only the http-client-csharp package build is needed. --- ## Step 1 — Discover coverage gaps ### How specs are organized The Azure C# emitter tests specs from **two sources**: 1. **Standard HTTP specs** (`@typespec/http-specs`): Located at `{PKG}/node_modules/@typespec/http-specs/specs/` 2. **Azure HTTP specs** (`@azure-tools/azure-http-specs`): Located at `{PKG}/node_modules/@azure-tools/azure-http-specs/specs/` ### How specs are filtered The file `{PKG}/eng/scripts/Spector-Helper.psm1` defines which specs are included/excluded. **Failing/excluded specs** are defined in the `$failingSpecs` array. Always check that file for the current list — do not hardcode specs here. Current exclusions fall into three categories: 1. **Standard spec issues**: `streaming/jsonl`, `response/status-code-range` (namespace conflict with `Azure.Response`), `type/file` 2. **Azure specs not yet buildable**: `azure/client-generator-core/alternate-type`, `azure/client-generator-core/deserialize-empty-string-as-null` 3. **Delegated to management generator**: All `azure/resource-manager/*` specs (common-properties, non-resource, operation-templates, resources, large-header, method-subscription-id, multi-service variants) ### Discovering gaps programmatically Compare available specs against existing tests. Run this from `{PKG}`: ```powershell Import-Module "{PKG}/eng/scripts/Spector-Helper.psm1" -DisableNameChecking -Force # Get all valid specs $specs = Get-Sorted-Specs | ForEach-Object { Get-SubPath $_ } # Get all test files $testFiles = Get-ChildItem -Path "{PKG}/generator/TestProjects/Spector.Tests/Http" -Recurse -Filter "*Tests.cs" | ForEach-Object { $_.FullName } # For each spec, check if a corresponding test directory/file exists foreach ($spec in $specs) { $specParts = $spec -replace '/', '\' -split '\\' $testPath = "{PKG}/generator/TestProjects/Spector.Tests/Http" # ... check if test exists } ``` ### Comparing Azure vs Standard dashboards The two dashboards report coverage for different scopes: - **Standard dashboard** (typespec.io): Shows coverage for `@typespec/http-specs` scenarios only, reported by the base `@typespec/http-client-csharp` emitter in the `microsoft/typespec` repo. - **Azure dashboard** (azure.github.io/typespec-azure): Shows coverage for **both** `@typespec/http-specs` and `@azure-tools/azure-http-specs` scenarios, reported by the `@azure-tools/typespec-csharp` emitter in this repo. **To find gaps between the two dashboards:** 1. Open both dashboards side-by-side. 2. For each Standard scenario shown as ✅ on typespec.io but ❌/missing on the Azure dashboard, that is a **regression or missing port** from the base emitter. 3. For each Standard scenario shown as ❌ on typespec.io but ✅ on the Azure dashboard, that is an **Azure-only enhancement**. 4. Azure-specific scenarios (`azure/*`) only appear on the Azure dashboard. Common causes of Standard coverage gaps in Azure: - The base emitter in `microsoft/typespec` added a test that hasn't been ported here yet. - The Azure emitter excludes a spec (check `$failingSpecs` in `Spector-Helper.psm1`). - Namespace or build conflicts specific to the Azure generator (e.g., `Response` vs `Azure.Response`). ### Existing test coverage **Standard spec test directories** (under `Spector.Tests/Http/`): ``` Authentication/ (ApiKey, Http/Custom, OAuth2, Union) Client/ (ClientNamespace, Naming, Overload, Structure/*) Encode/ (Array, Bytes, DateTime, Duration, Numeric) Parameters/ (Basic, BodyOptionality, CollectionFormat, Path, Query, Spread) Payload/ (ContentNegotiation, JsonMergePatch, MediaType, MultiPart, Pageable, Xml) Resiliency/ (SrvDriven/V1, SrvDriven/V2) Routes/ Serialization/ (EncodedName/Json) Server/ (Endpoint/NotDefined, Path/Multiple, Path/Single, Versions/*) Service/ SpecialHeaders/ (ConditionalRequest, Repeatability) SpecialWords/ Versioning/ (Added, MadeOptional, Removed, RenamedFrom, ReturnTypeChangedFrom, TypeChangedFrom) _Type/ (Array, Dictionary, Enum/*, Model/*, Property/*, Scalar, Union) ``` **Azure spec test directories** (under `Spector.Tests/Http/Azure/`): ``` ClientGeneratorCore/ (Access, ApiVersion/*, ClientInitialization/*, ClientLocation/*, HierarchyBuilding, Override, Usage) Core/ (Basic, Lro/Rpc, Lro/Standard, Model, Page, Scalar, Traits) Encode/ (Duration) Example/ (Basic) Payload/ (Pageable) SpecialHeaders/ (ClientRequestId) Versioning/ (PreviewVersion/V1, PreviewVersion/V2) ``` ### Accurate gap detection The naive approach of matching spec paths to test directory names can produce false positives because: - C# reserved words get underscore-prefixed in test dirs (e.g., `type/` → `_Type/`, `array` → `_Array`, `enum` → `_Enum`) - kebab-case gets converted to PascalCase (e.g., `content-negotiation` → `ContentNegotiation`) - Some test files cover a parent spec but not sub-specs (e.g., `UnionTests.cs` covers `type/union` but NOT `type/union/discriminated`) **To find real gaps**, verify each candidate by checking whether a test _directory_ exists for the spec's exact path, including sub-paths. A test file at a parent level does NOT cover child specs. > **Important:** The gap list evolves over time. Always re-run the comparison to get current gaps. All committed Spector libraries are stubbed — `[SpectorTest]` will auto-skip tests unless you regenerate with `-Stubbed $false` or use `Test-Spector.ps1`. ### Understanding stubbed vs unstubbed generation All Spector libraries **committed to the repository are stubbed**. `Generate.ps1` defaults to `$Stubbed = $true`, which causes the emitter to use `AzureStubGenerator` instead of `AzureClientGenerator`. Stubbed clients use expression-bodied constructors (`=>`) instead of block bodies (`{ }`), and the `[SpectorTest]` attribute automatically skips tests for stubbed clients. To generate **unstubbed** code (for local testing), pass `-Stubbed $false`: ```powershell pwsh eng/scripts/Generate.ps1 -filter "<spec-path>" -Stubbed $false ``` However, the recommended way to test is via `Test-Spector.ps1` (see Step 7), which handles the unstubbed regeneration, test execution, and directory restoration automatically. --- ## Step 2 — Read the spec to understand expected behavior Spec files live in: - **Standard**: `{PKG}/node_modules/@typespec/http-specs/specs/` - **Azure**: `{PKG}/node_modules/@azure-tools/azure-http-specs/specs/` If `node_modules` is not installed, specs can also be found in the upstream repos: - <https://github.com/microsoft/typespec/tree/main/packages/http-specs/specs> - <https://github.com/Azure/typespec-azure/tree/main/packages/azure-http-specs/specs> Each spec contains: - **`main.tsp`** — the TypeSpec definition with `@scenario` and `@scenarioDoc` decorators - **`client.tsp`** (optional) — client-level customizations; takes priority over `main.tsp` during generation - **`tspconfig.yaml`** (optional in the Spector test project) — C#-specific generation options Read the `@scenarioDoc` decorators to understand: - The HTTP method, path, and expected parameters - The expected request body shape - The expected response status code and body - Any special behavior (e.g., "should return 204", "should send header X") --- ## Step 3 — Generate the C# client for the target spec Use `Generate.ps1` with a filter to generate only the specific spec: ```powershell cd {PKG} # Generate unstubbed (for local testing/development) pwsh eng/scripts/Generate.ps1 -filter "<spec-path>" -Stubbed $false # Generate stubbed (default, matches what is committed to repo) pwsh eng/scripts/Generate.ps1 -filter "<spec-path>" ``` Examples: ```powershell # Standard spec (unstubbed for testing) pwsh eng/scripts/Generate.ps1 -filter "http/encode/duration" -Stubbed $false # Azure spec pwsh eng/scripts/Generate.ps1 -filter "http/azure/core/lro" -Stubbed $false # Versioning spec (generates v1 + v2 automatically) pwsh eng/scripts/Generate.ps1 -filter "http/versioning/added" -Stubbed $false ``` The generated code lands in `{PKG}/generator/TestProjects/Spector/<spec-path>/src/Generated/`. ### Verify generation succeeded ```powershell # Check that client code was generated Get-ChildItem "{PKG}/generator/TestProjects/Spector/<spec-path>/src/Generated/" -Filter "*Client.cs" ``` > **Note:** All committed Spector libraries are stubbed. The `[SpectorTest]` attribute automatically skips tests for stubbed clients. Use `Test-Spector.ps1` (Step 7) to regenerate unstubbed, run tests, and restore automatically. --- ## Step 4 — Read the generated client API surface Browse the generated code to understand: 1. **Client class(es)**: `*Client.cs` — the entry point(s) 2. **Sub-clients**: accessed via `Get*Client()` methods 3. **Operations**: async methods like `GetAsync()`, `PutAsync(body)`, `SendAsync()` 4. **Models**: under `Models/` — request/response shapes 5. **Constructor signature**: `new XClient(Uri endpoint, XClientOptions options)` or `new XClient(Uri endpoint, KeyCredential credential, XClientOptions options)` Pay attention to: - Method names (they map to TypeSpec operation names) - Parameter types (models, primitives, BinaryData, RequestContent) - Return types (`ClientResult`, `ClientResult<T>`, `AsyncPageable<T>`) - **Azure-specific patterns**: `RequestContent` usage, `Azure.Core` types, internal vs public visibility --- ## Step 5 — Create or extend the test file ### Directory and namespace conventions The test directory structure mirrors the spec path with these transformations: - `http/` → `Http/` - kebab-case → PascalCase (e.g., `content-negotiation` → `ContentNegotiation`) - `type/` → `_Type/` (leading underscore because `Type` is a C# keyword) - `array` → `_Array` (same reason) - `enum` → `_Enum` (same reason) - `azure/` → `Azure/` (Azure specs get their own top-level test directory) **Namespace pattern**: `TestProjects.Spector.Tests.Http.<PascalCasePath>` Example mappings: | Spec path | Test directory | Namespace | |-----------|---------------|-----------| | `http/encode/duration` | `Http/Encode/Duration/` | `TestProjects.Spector.Tests.Http.Encode.Duration` |