cli-e2e-testing

# Aspire CLI End-to-End Testing with Hex1b This skill provides patterns and practices for writing end-to-end tests for the Aspire CLI using the Hex1b terminal automation library. ## Overview CLI E2E tests use the Hex1b library to automate terminal sessions, simulating real user interactions with the Aspire CLI. Tests run in CI with asciinema recordings for debugging. **Location**: `tests/Aspire.Cli.EndToEnd.Tests/` **Supported Platforms**: Linux only. Hex1b requires a Linux terminal environment. Tests are configured to skip on Windows and macOS in CI. ## Key Components ### Core Classes - **`Hex1bTerminal`**: The main terminal class from the Hex1b library for terminal automation - **`Hex1bTerminalAutomator`**: Async/await API for driving a `Hex1bTerminal` — the preferred approach for new tests - **`Hex1bAutomatorTestHelpers`** (shared helpers): Async extension methods on `Hex1bTerminalAutomator` (`WaitForSuccessPromptAsync`, `AspireNewAsync`, etc.) - **`CliE2EAutomatorHelpers`** (`Helpers/CliE2EAutomatorHelpers.cs`): CLI-specific async extension methods on `Hex1bTerminalAutomator` (`PrepareDockerEnvironmentAsync`, `InstallAspireCliAsync`, etc.) - **`CellPatternSearcher`**: Pattern matching for terminal cell content - **`SequenceCounter`** (`Helpers/SequenceCounter.cs`): Tracks command execution count for deterministic prompt detection - **`CliE2ETestHelpers`** (`Helpers/CliE2ETestHelpers.cs`): Environment variable helpers and terminal factory methods - **`TemporaryWorkspace`**: Creates isolated temporary directories for test execution - **`Hex1bTerminalInputSequenceBuilder`** *(legacy)*: Fluent builder API for building sequences of terminal input/output operations. Prefer `Hex1bTerminalAutomator` for new tests. ### Test Architecture Each test: 1. Creates a `TemporaryWorkspace` for isolation 2. Builds a `Hex1bTerminal` with headless mode and asciinema recording 3. Creates a `Hex1bTerminalAutomator` wrapping the terminal 4. Drives the terminal with async/await calls and awaits completion ## Test Structure ```csharp public sealed class SmokeTests(ITestOutputHelper output) { [Fact] public async Task MyCliTest() { var repoRoot = CliE2ETestHelpers.GetRepoRoot(); var strategy = CliInstallStrategy.Detect(output.WriteLine); var workspace = TemporaryWorkspace.Create(output); using var terminal = CliE2ETestHelpers.CreateDockerTestTerminal(repoRoot, strategy, output, workspace: workspace); var counter = new SequenceCounter(); var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500)); await using var terminalRun = CliE2ETestHelpers.StartRun(terminal, workspace, auto, counter, output, TestContext.Current.CancellationToken); await auto.PrepareDockerEnvironmentAsync(counter, workspace); await auto.InstallAspireCliAsync(strategy, counter); await auto.TypeAsync("aspire --version"); await auto.EnterAsync(); await auto.WaitForSuccessPromptAsync(counter); } } ``` ### TerminalRun Pattern **Always use `CliE2ETestHelpers.StartRun`** to wrap the terminal run. This returns a `TerminalRun` (implements `IAsyncDisposable`) that automatically: 1. Captures Aspire diagnostics via `CaptureAspireDiagnosticsAsync` (best effort) 2. Types `exit` and presses Enter to close the terminal 3. Awaits the pending run task This eliminates the need for manual `exit`/`await pendingRun` at the end of every test and ensures diagnostics are always captured, even when tests fail. ```csharp // DO: Use StartRun for consistent diagnostics capture and cleanup using var terminal = CliE2ETestHelpers.CreateDockerTestTerminal(repoRoot, strategy, output, workspace: workspace); var counter = new SequenceCounter(); var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500)); await using var terminalRun = CliE2ETestHelpers.StartRun(terminal, workspace, auto, counter, output, TestContext.Current.CancellationToken); // ... test body — no exit/pendingRun needed at the end // DON'T: Manually handle exit and pendingRun var pendingRun = terminal.RunAsync(TestContext.Current.CancellationToken); // ... test body ... await auto.TypeAsync("exit"); await auto.EnterAsync(); await pendingRun; ``` ## Running Tests Locally CLI E2E tests run inside Docker containers on Linux. The workflow is: build a portable archive with `localhive`, then point the tests at it. This is the primary way to iterate on E2E tests during development. ### Prerequisites - Docker Desktop running (with Linux containers) - .NET 10 SDK (installed via `./restore.sh` or `.\restore.cmd`) ### Quick Start (macOS / Linux) ```bash # 1. Build a portable archive with CLI + packages + bundle # Use linux-arm64 on Apple Silicon, linux-x64 on Intel/Linux ./localhive.sh -o /tmp/aspire-e2e -r linux-arm64 --archive # 2. Run a specific test ASPIRE_E2E_ARCHIVE=/tmp/aspire-e2e.tar.gz \ dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \ -- --filter-method "*.CreateAndRunAspireStarterProject" # 3. Run all tests in a class ASPIRE_E2E_ARCHIVE=/tmp/aspire-e2e.tar.gz \ dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \ -- --filter-class "*.SmokeTests" ``` ### Quick Start (Windows / PowerShell) ```powershell # 1. Build a portable archive (Docker Desktop uses linux-x64 via WSL2) .\localhive.ps1 -o C:\tmp\aspire-e2e -r linux-x64 -Archive # 2. Run a specific test $env:ASPIRE_E2E_ARCHIVE = "C:\tmp\aspire-e2e.tar.gz" dotnet test tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj ` -- --filter-method "*.CreateAndRunAspireStarterProject" # 3. Run all tests in a class dotnet test tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj ` -- --filter-class "*.SmokeTests" ``` ### Choosing the Right RID The archive must match the Docker container's architecture: | Host | Docker Desktop | RID | |------|---------------|-----| | Apple Silicon Mac | Linux arm64 containers | `linux-arm64` | | Intel Mac | Linux x64 containers | `linux-x64` | | Windows (any) | WSL2 Linux x64 | `linux-x64` | | Linux x64 | Native | `linux-x64` | | Linux arm64 | Native | `linux-arm64` | ### Development Workflow The typical loop when writing or debugging E2E tests: ```bash # 1. Make your code changes (CLI, hosting, templates, etc.) # 2. Rebuild the archive (picks up all changes — ~3 min) ./localhive.sh -o /tmp/aspire-e2e -r linux-arm64 --archive # 3. Run the specific test you're working on ASPIRE_E2E_ARCHIVE=/tmp/aspire-e2e.tar.gz \ dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \ -- --filter-method "*.YourTestName" # 4. If it fails, check the asciinema recording # Recordings are saved under the test output TestResults/recordings/ directory # Play with: asciinema play /path/to/YourTestName.cast # 5. Fix and repeat from step 1 or 2 ``` ### Install Modes The `CliInstallStrategy` class auto-detects how to install the CLI in the test container. You can override via environment variables: | Env Var | Mode | Example | |---------|------|---------| | `ASPIRE_E2E_ARCHIVE` | LocalHive — extract archive into container | `/tmp/aspire-e2e.tar.gz` | | `ASPIRE_E2E_QUALITY` | Install script with quality | `dev`, `staging`, `release` | | `ASPIRE_E2E_VERSION` | Install script with version | `13.2.1` | | *(none, in CI)* | PullRequest — install from PR artifacts | Auto-detected | | *(none, locally)* | InstallScript (latest GA) | Auto-detected | **LocalHive** (via `ASPIRE_E2E_ARCHIVE`) is the recommended mode for local development — it uses your locally-built CLI, packages, and bundle so you test exactly what you've changed. ### Testing Against Released Versions Useful for verifying tests pass against shipped versions or catching regressions: ```bash # Test against latest GA release ASPIRE_E2E_QUALITY=release dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \ -- --filter-method "*.CreateAndRunAspireStarterProject" # Test against daily builds ASPIRE_E2E_QUALITY=dev dotnet test ... # Test against a specific version ASPIRE_E2E_VERSION=13.2.1 dotnet test ... ``` ### Emulated channel matrix tests (identity sidecar) A set of tests validates the **CLI identity sidecar** — the ability to make a locally built CLI emulate a different channel/version via `ASPIRE_CLI_*` env vars. They form an AppHost-language × channel-emulation matrix (one test per language because C# and TypeScript scaffold through different code paths and have diverged before): | Class | Channel emulated | `Aspire*` source | NuGet.config dropped? | |-------|------------------|------------------|------------------------| | `EmulatedReleasedBuildTests` | `stable` (latest shipped) | nuget.org | No (C# and TS) | | `EmulatedStagingBuildTests` | `staging` (latest darc build) | `darc-pub-...` feed | Yes — darc feed pin | | `EmulatedLocalReleaseBuildTests` | `stable` (**future, local-only**) | local hive via `ASPIRE_CLI_PACKAGES` | No (C# and TS) | `EmulatedLocalReleaseBuildTests` is the **all-local "future release"** row: it emulates a version (e.g. `13.5.0`) that exists only in a locally built hive, so a successful resolve proves the CLI consulted `ASPIRE_CLI_PACKAGES` rather than nuget.org. Run it by building a **stable-shaped** archive with `localhive --version`: ```bash # 1. Build a stable-shaped archive (note: --version X.Y.Z, NOT a prerelease suffix) ./localhive.sh --version 13.5.0 -o /tmp/aspire-localrelease -r linux-arm64 --archive # 2. Run the all-local class ASPIRE_E2E_ARCHIVE=/tmp/aspire-localrelease.tar.gz \ dotnet test --project tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \ -- --filter-class "*.EmulatedLocalReleaseBuildTests" ``` These tests **skip** unless the CLI was installed from a `LocalHive` archive *and* that archive is stable-shaped (no prerelease suffix). In default CI the archive is a prerelease `LocalArchive`, so they skip and add zero CI cost — CI relies on `--ignore-exit-code 8` (set in `eng/Testing.targets` `MtpBaseArgs`) so an all-skipped class job still passes. The test also registers the hive as an **ambient** NuGet source (`dotnet nuget add source`) because MSBuild resolves the apphost's `Aspire.AppHost.Sdk` before restore from nuget.config sources only, ignoring `ASPIRE_CLI_PACKAGES`. > **⚠️ Local rebuilds: isolate the NuGet global cache.** This only affects **local** iteration of a > stable-shaped emulation (the E2E tests run in fresh Docker containers, so CI is immune). NuGet's > global packages folder (`~/.nuget/packages/<id>/<version>/`) caches **extracted** packages keyed by > version. When you emulate a *fixed* stable version (e.g. `13.5.0`) and **rebuild** it, a stale > `13.5.0` in that shared cache silently shadows the freshly built one — same version, different > content — so restore drifts (the stale AppHost SDK injects a prerelease floor and you get `NU1603` > warnings binding the graph to a stray `13.5.0-pr.…`). **Fix:** point `NUGET_PACKAGES` at a > per-emulation directory (`export NUGET_PACKAGES=/tmp/aspire-localrelease/.nuget-packages`). > `localhive … -o DIR`'s generated `activate.sh`/`activate.ps1` already sets this up. See the cache > hazard note in `.agents/skills/cli-channel-debugging/SKILL.md` (Scenario 7c) for the full mechanism. ## SequenceCounter and Prompt Detection The `SequenceCounter` class tracks the number of shell commands executed. This enables deterministic waiting for command completion via a custom shell prompt. ### How It Works 1. `PrepareDockerEnvironmentAsync()` configures the shell with a custom prompt: `[N OK] $ ` or `[N ERR:code] $ ` 2. Each command increments the counter 3. `WaitForSuccessPromptAsync(counter)` waits for a prompt showing the current count with `OK` ```csharp var counter = new SequenceCounter(); var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500)); await auto.PrepareDockerEnvironmentAsync(counter, workspace); // Sets up prompt, counter starts at 1 await auto.TypeAsync("echo hello"); await auto.EnterAsync(); await auto.WaitForSuccessPromptAsync(counter); // Waits for "[1 OK] $ ", then increments to 2 await auto.TypeAsync("ls -la"); await auto.EnterAsync(); await auto.WaitForSuccessPromptAsync(counter); // Waits for "[2 OK] $ ", then increments to 3 await auto.TypeAsync("exit"); await auto.EnterAsync(); ``` This approach is more reliable than arbitrary timeouts because it deterministically waits for each command to complete. ## Pattern Searching with CellPatternSearcher Use `CellPatternSearcher` to find text patterns in terminal output: ```csharp // Simple text search (literal string matching - PREFERRED) var waitingForPrompt = new CellPatternSearcher() .Find("Enter the project name"); // Literal string with special characters (use Find, not FindPattern!) var waitingForTemplate = new CellPatternSearcher() .Find("> Starter App (FastAPI/React)"); // Parentheses and slashes are literal // Regex pattern (only when you need wildcards/regex features) var waitingForAnyStarter = new CellPatternSearcher() .FindPattern("> Starter App.*"); // .* matches anything // Chained patterns (find "b", then scan right until "$", then right of " ") var waitingForShell = new CellPatternSearcher() .Find("b").RightUntil("$").Right(' ').Right(' '); // Use in WaitUntilAsync await auto.WaitUntilAsync( snapshot => waitingForPrompt.Search(snapshot).Count > 0, TimeSpan.FromSeconds(30), description: "waiting for prompt"); ``` ### Find vs FindPattern - **`Find(string)`**: Literal string matching. Use this for most cases. - **`FindPattern(string)`**: Regex pattern matching. Use only when you need regex features like wildcards. **Important**: If your search string contains regex special characters like `(`, `)`, `/`, `.`, `*`, `+`, `?`, `[`, `]`, `{`, `}`, `^`, `$`, `|`, or `\`, use `Find()` instead of `FindPattern()` to avoid regex interpretation.