cli-e2e-testing

# Aspire CLI End-to-End Testing with Hex1b This skill provides patterns and practices for writing end-to-end tests for the Aspire CLI using the Hex1b terminal automation library. ## Overview CLI E2E tests use the Hex1b library to automate terminal sessions, simulating real user interactions with the Aspire CLI. Tests run in CI with asciinema recordings for debugging. **Location**: `tests/Aspire.Cli.EndToEnd.Tests/` **Supported Platforms**: Linux only. Hex1b requires a Linux terminal environment. Tests are configured to skip on Windows and macOS in CI. ## Key Components ### Core Classes - **`Hex1bTerminal`**: The main terminal class from the Hex1b library for terminal automation - **`Hex1bTerminalAutomator`**: Async/await API for driving a `Hex1bTerminal` — the preferred approach for new tests - **`Hex1bAutomatorTestHelpers`** (shared helpers): Async extension methods on `Hex1bTerminalAutomator` (`WaitForSuccessPromptAsync`, `AspireNewAsync`, etc.) - **`CliE2EAutomatorHelpers`** (`Helpers/CliE2EAutomatorHelpers.cs`): CLI-specific async extension methods on `Hex1bTerminalAutomator` (`PrepareDockerEnvironmentAsync`, `InstallAspireCliAsync`, etc.) - **`CellPatternSearcher`**: Pattern matching for terminal cell content - **`SequenceCounter`** (`Helpers/SequenceCounter.cs`): Tracks command execution count for deterministic prompt detection - **`CliE2ETestHelpers`** (`Helpers/CliE2ETestHelpers.cs`): Environment variable helpers and terminal factory methods - **`TemporaryWorkspace`**: Creates isolated temporary directories for test execution - **`Hex1bTerminalInputSequenceBuilder`** *(legacy)*: Fluent builder API for building sequences of terminal input/output operations. Prefer `Hex1bTerminalAutomator` for new tests. ### Test Architecture Each test: 1. Creates a `TemporaryWorkspace` for isolation 2. Builds a `Hex1bTerminal` with headless mode and asciinema recording 3. Creates a `Hex1bTerminalAutomator` wrapping the terminal 4. Drives the terminal with async/await calls and awaits completion ## Test Structure ```csharp public sealed class SmokeTests(ITestOutputHelper output) { [Fact] public async Task MyCliTest() { var repoRoot = CliE2ETestHelpers.GetRepoRoot(); var strategy = CliInstallStrategy.Detect(output.WriteLine); var workspace = TemporaryWorkspace.Create(output); using var terminal = CliE2ETestHelpers.CreateDockerTestTerminal(repoRoot, strategy, output, workspace: workspace); var counter = new SequenceCounter(); var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500)); await using var terminalRun = CliE2ETestHelpers.StartRun(terminal, workspace, auto, counter, output, TestContext.Current.CancellationToken); await auto.PrepareDockerEnvironmentAsync(counter, workspace); await auto.InstallAspireCliAsync(strategy, counter); await auto.TypeAsync("aspire --version"); await auto.EnterAsync(); await auto.WaitForSuccessPromptAsync(counter); } } ``` ### TerminalRun Pattern **Always use `CliE2ETestHelpers.StartRun`** to wrap the terminal run. This returns a `TerminalRun` (implements `IAsyncDisposable`) that automatically: 1. Captures Aspire diagnostics via `CaptureAspireDiagnosticsAsync` (best effort) 2. Types `exit` and presses Enter to close the terminal 3. Awaits the pending run task This eliminates the need for manual `exit`/`await pendingRun` at the end of every test and ensures diagnostics are always captured, even when tests fail. ```csharp // DO: Use StartRun for consistent diagnostics capture and cleanup using var terminal = CliE2ETestHelpers.CreateDockerTestTerminal(repoRoot, strategy, output, workspace: workspace); var counter = new SequenceCounter(); var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500)); await using var terminalRun = CliE2ETestHelpers.StartRun(terminal, workspace, auto, counter, output, TestContext.Current.CancellationToken); // ... test body — no exit/pendingRun needed at the end // DON'T: Manually handle exit and pendingRun var pendingRun = terminal.RunAsync(TestContext.Current.CancellationToken); // ... test body ... await auto.TypeAsync("exit"); await auto.EnterAsync(); await pendingRun; ``` ## Running Tests Locally CLI E2E tests run inside Docker containers on Linux. The workflow is: build a portable archive with `localhive`, then point the tests at it. This is the primary way to iterate on E2E tests during development. ### Prerequisites - Docker Desktop running (with Linux containers) - .NET 10 SDK (installed via `./restore.sh` or `.\restore.cmd`) ### Quick Start (macOS / Linux) ```bash # 1. Build a portable archive with CLI + packages + bundle # Use linux-arm64 on Apple Silicon, linux-x64 on Intel/Linux ./localhive.sh -o /tmp/aspire-e2e -r linux-arm64 --archive # 2. Run a specific test ASPIRE_E2E_ARCHIVE=/tmp/aspire-e2e.tar.gz \ dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \ -- --filter-method "*.CreateAndRunAspireStarterProject" # 3. Run all tests in a class ASPIRE_E2E_ARCHIVE=/tmp/aspire-e2e.tar.gz \ dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \ -- --filter-class "*.SmokeTests" ``` ### Quick Start (Windows / PowerShell) ```powershell # 1. Build a portable archive (Docker Desktop uses linux-x64 via WSL2) .\localhive.ps1 -o C:\tmp\aspire-e2e -r linux-x64 -Archive # 2. Run a specific test $env:ASPIRE_E2E_ARCHIVE = "C:\tmp\aspire-e2e.tar.gz" dotnet test tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj ` -- --filter-method "*.CreateAndRunAspireStarterProject" # 3. Run all tests in a class dotnet test tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj ` -- --filter-class "*.SmokeTests" ``` ### Choosing the Right RID The archive must match the Docker container's architecture: | Host | Docker Desktop | RID | |------|---------------|-----| | Apple Silicon Mac | Linux arm64 containers | `linux-arm64` | | Intel Mac | Linux x64 containers | `linux-x64` | | Windows (any) | WSL2 Linux x64 | `linux-x64` | | Linux x64 | Native | `linux-x64` | | Linux arm64 | Native | `linux-arm64` | ### Development Workflow The typical loop when writing or debugging E2E tests: ```bash # 1. Make your code changes (CLI, hosting, templates, etc.) # 2. Rebuild the archive (picks up all changes — ~3 min) ./localhive.sh -o /tmp/aspire-e2e -r linux-arm64 --archive # 3. Run the specific test you're working on ASPIRE_E2E_ARCHIVE=/tmp/aspire-e2e.tar.gz \ dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \ -- --filter-method "*.YourTestName" # 4. If it fails, check the asciinema recording # Recordings are saved under the test output TestResults/recordings/ directory # Play with: asciinema play /path/to/YourTestName.cast # 5. Fix and repeat from step 1 or 2 ``` ### Install Modes The `CliInstallStrategy` class auto-detects how to install the CLI in the test container. You can override via environment variables: | Env Var | Mode | Example | |---------|------|---------| | `ASPIRE_E2E_ARCHIVE` | LocalHive — extract archive into container | `/tmp/aspire-e2e.tar.gz` | | `ASPIRE_E2E_QUALITY` | Install script with quality | `dev`, `staging`, `release` | | `ASPIRE_E2E_VERSION` | Install script with version | `13.2.1` | | *(none, in CI)* | PullRequest — install from PR artifacts | Auto-detected | | *(none, locally)* | InstallScript (latest GA) | Auto-detected | **LocalHive** (via `ASPIRE_E2E_ARCHIVE`) is the recommended mode for local development — it uses your locally-built CLI, packages, and bundle so you test exactly what you've changed. ### Testing Against Released Versions Useful for verifying tests pass against shipped versions or catching regressions: ```bash # Test against latest GA release ASPIRE_E2E_QUALITY=release dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \ -- --filter-method "*.CreateAndRunAspireStarterProject" # Test against daily builds ASPIRE_E2E_QUALITY=dev dotnet test ... # Test against a specific version ASPIRE_E2E_VERSION=13.2.1 dotnet test ... ``` ## SequenceCounter and Prompt Detection The `SequenceCounter` class tracks the number of shell commands executed. This enables deterministic waiting for command completion via a custom shell prompt. ### How It Works 1. `PrepareDockerEnvironmentAsync()` configures the shell with a custom prompt: `[N OK] $ ` or `[N ERR:code] $ ` 2. Each command increments the counter 3. `WaitForSuccessPromptAsync(counter)` waits for a prompt showing the current count with `OK` ```csharp var counter = new SequenceCounter(); var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500)); await auto.PrepareDockerEnvironmentAsync(counter, workspace); // Sets up prompt, counter starts at 1 await auto.TypeAsync("echo hello"); await auto.EnterAsync(); await auto.WaitForSuccessPromptAsync(counter); // Waits for "[1 OK] $ ", then increments to 2 await auto.TypeAsync("ls -la"); await auto.EnterAsync(); await auto.WaitForSuccessPromptAsync(counter); // Waits for "[2 OK] $ ", then increments to 3 await auto.TypeAsync("exit"); await auto.EnterAsync(); ``` This approach is more reliable than arbitrary timeouts because it deterministically waits for each command to complete. ## Pattern Searching with CellPatternSearcher Use `CellPatternSearcher` to find text patterns in terminal output: ```csharp // Simple text search (literal string matching - PREFERRED) var waitingForPrompt = new CellPatternSearcher() .Find("Enter the project name"); // Literal string with special characters (use Find, not FindPattern!) var waitingForTemplate = new CellPatternSearcher() .Find("> Starter App (FastAPI/React)"); // Parentheses and slashes are literal // Regex pattern (only when you need wildcards/regex features) var waitingForAnyStarter = new CellPatternSearcher() .FindPattern("> Starter App.*"); // .* matches anything // Chained patterns (find "b", then scan right until "$", then right of " ") var waitingForShell = new CellPatternSearcher() .Find("b").RightUntil("$").Right(' ').Right(' '); // Use in WaitUntilAsync await auto.WaitUntilAsync( snapshot => waitingForPrompt.Search(snapshot).Count > 0, TimeSpan.FromSeconds(30), description: "waiting for prompt"); ``` ### Find vs FindPattern - **`Find(string)`**: Literal string matching. Use this for most cases. - **`FindPattern(string)`**: Regex pattern matching. Use only when you need regex features like wildcards. **Important**: If your search string contains regex special characters like `(`, `)`, `/`, `.`, `*`, `+`, `?`, `[`, `]`, `{`, `}`, `^`, `$`, `|`, or `\`, use `Find()` instead of `FindPattern()` to avoid regex interpretation. ## Extension Methods ### Hex1bAutomatorTestHelpers Extensions (Shared — Automator API) | Method | Description | |--------|-------------| | `WaitForSuccessPromptAsync(counter, timeout?)` | Waits for `[N OK] $ ` prompt, fails immediately if error prompt appears, and increments counter | | `WaitForAnyPromptAsync(counter, timeout?)` | Waits for any prompt (`OK` or `ERR`) and increments counter | | `WaitForErrorPromptAsync(counter, timeout?)` | Waits for `[N ERR:code] $ ` prompt and increments counter | | `RunCommandAsync(command, counter, timeout?)` | Types a command, presses Enter, and waits for success prompt (fails fast on error) | | `DeclineAgentInitPromptAsync()` | Declines the `aspire agent init` prompt if it appears | | `AspireNewAsync(projectName, counter, template?, useRedisCache?)` | Runs `aspire new` interactively, handling template selection, project name, output path, URLs, Redis, and test project prompts | See [AspireNew Helper](#aspirenew-helper) below for detailed usage. ### CliE2EAutomatorHelpers Extensions on Hex1bTerminalAutomator | Method | Description | |--------|-------------| | `PrepareDockerEnvironmentAsync(counter, workspace)` | Sets up Docker container environment with custom prompt and command tracking | | `InstallAspireCliAsync(installMode, counter)` | Installs the Aspire CLI inside the Docker container | | `ClearScreenAsync(counter)` | Clears the terminal screen and waits for prompt | ### SequenceCounterExtensions | Method | Description | |--------|-------------| | `IncrementSequence(counter)` | Manually increments the counter | ### Legacy Builder Extensions The following extensions on `Hex1bTerminalInputSequenceBuilder` are still available but should not be used in new tests: | Method | Description | |--------|-------------| | `WaitForSuccessPrompt(counter, timeout?)` | *(legacy)* Waits for `[N OK] $ ` prompt and increments counter | | `PrepareEnvironment(workspace, counter)` | *(legacy)* Sets up custom prompt with command tracking | | `SourceAspireBundleEnvironment(counter)` | *(legacy)* Sources bundle PATH environment variables | ## DO: Use CellPatternSearcher for Output Detection Wait for specific output patterns rather than arbitrary delays: ```csharp var waitingForMessage = new CellPatternSearcher() .Find("Project created successfully."); await auto.TypeAsync("aspire new"); await auto.EnterAsync(); await auto.WaitUntilAsync( s => waitingForMessage.Search(s).Count > 0, TimeSpan.FromMinutes(2), description: "waiting for project created message"); ``` ## DO: Use WaitForSuccessPromptAsync After Commands After running shell commands, use `WaitForSuccessPromptAsync()` to wait for the command to complete: ```csharp await auto.TypeAsync("dotnet build"); await auto.EnterAsync(); await auto.WaitForSuccessPromptAsync(counter); // Waits for prompt, verifies success await auto.TypeAsync("dotnet run"); await auto.EnterAsync(); await auto.WaitForSuccessPromptAsync(counter); ``` ## AspireNew Helper The `AspireNew` extension method centralizes the multi-step `aspire new` interactive flow. Use it instead of manually building the prompt sequence. ### AspireTemplate Enum | Value | Template | Arrow Keys |