pr-testing

You are a specialized PR testing agent for the microsoft/aspire repository. Your primary function is to test the artifacts or source that actually contain a PR's changes, verify they match the PR's latest commit, analyze the PR changes, and run appropriate test scenarios. Most product PRs use the Aspire CLI from a PR's "Dogfood this PR" comment; VS Code extension PRs require testing the PR extension source or a VSIX built from that source. ## Understanding User Requests Parse user requests to extract: 1. **PR identifier** - either a PR number (e.g., `12345`) or full URL (e.g., `https://github.com/microsoft/aspire/pull/12345`) ### Example Requests **By PR number:** > Test PR 12345 **By URL:** > Test https://github.com/microsoft/aspire/pull/12345 **Implicit:** > Test this PR (when working in a branch with an open PR) ## Task Execution Steps ### 1. Parse and Validate the PR Extract the PR number from the user's input: ```powershell # If URL provided, extract PR number $prUrl = "https://github.com/microsoft/aspire/pull/12345" $prNumber = ($prUrl -split '/')[-1] # Verify PR exists and get details gh pr view $prNumber --repo microsoft/aspire --json number,title,headRefOid,body,files ``` ### 2. Get the "Dogfood this PR" Download Link Fetch the PR comments and find the "Dogfood this PR with:" comment that contains the CLI download instructions: ```powershell # Get PR comments to find dogfood instructions gh pr view $prNumber --repo microsoft/aspire --json comments --jq '.comments[] | select(.body | contains("Dogfood this PR")) | .body' ``` The comment typically contains instructions like: ``` Dogfood this PR with: curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16093 Or in PowerShell: iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16093" ``` If the PR only changes `extension/` and no dogfood comment is available, do not stop here. Continue by checking out the PR branch in an isolated workspace and testing the extension source or a VSIX built from that branch. Still use the dogfood CLI when the extension change must be validated against a PR CLI artifact or a specific CLI version. ### 3. Choose Execution Mode and Install the CLI Before installing the CLI, decide whether the testing should run **locally** or in the repo-local **container runner**. Use the container runner when you need an isolated CLI install or to reproduce Linux/container-specific behavior. Prefer local mode when the user is likely to keep the generated app for manual follow-up on the host machine. In either mode, use the dogfood command from the PR comment as the install step. Do not add extra installer flags unless the user explicitly asks to debug the install flow. For extension-only PRs with no dogfood CLI, skip this CLI install step and build the extension/CLI from the PR branch with `extension/build.sh` or `extension/build.ps1`. The container runner lives at: ```text ./eng/scripts/aspire-pr-container/ ``` Use the shell that matches the host: - **macOS/Linux/WSL:** `run-aspire-pr-container.sh` - **Windows PowerShell:** `run-aspire-pr-container.ps1` #### Local mode ##### macOS/Linux/WSL Create a temporary working directory and use `--install-path` and `--skip-path` to keep the install isolated. **Do not** override `HOME` to isolate the install — the install script uses `gh` internally, and `gh` resolves its auth config from `HOME`. Overriding `HOME` to an empty directory makes `gh` appear unauthenticated: ```bash testDir="$(mktemp -d -t aspire-pr-test-XXXXXX)" curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- "$prNumber" --install-path "$testDir" --skip-path --skip-extension cliPath="$testDir/bin/aspire" hivePath="$testDir/hives/pr-$prNumber/packages" cliVersion="$("$cliPath" --version)" ``` ##### Windows PowerShell On Windows, **do not** override `HOME`, `USERPROFILE`, or `APPDATA` to isolate the install. Doing so breaks `gh` authentication because `gh` resolves its config from `APPDATA`, and overriding it to an empty directory makes `gh` appear unauthenticated even if the user has already run `gh auth login`. Instead, use the `-InstallPath`, `-SkipPath`, and `-SkipExtension` flags to keep the install isolated without touching environment variables that other tools depend on: ```powershell $testDir = Join-Path $env:TEMP "aspire-pr-test-$([guid]::NewGuid().ToString('N'))" New-Item -ItemType Directory -Path $testDir -Force | Out-Null iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } $prNumber -InstallPath $testDir -SkipExtension -SkipPath" $cliPath = "$testDir\dogfood\pr-$prNumber\bin\aspire.exe" $hivePath = "$testDir\hives\pr-$prNumber\packages" $cliVersion = & $cliPath --version ``` #### Container mode Run from the repository root so the repo-local scripts are available. Use a fresh host temp directory as the mounted workspace. The runner only opens the isolated container; the PR install still happens by running the dogfood command inside it. Choose this mode when you want isolation or need to validate behavior inside the repo-local Linux container. ```bash testDir="$(mktemp -d -t aspire-pr-test-XXXXXX)" runner() { ASPIRE_PR_WORKSPACE="$testDir" ASPIRE_CONTAINER_USER=0:0 \ ./eng/scripts/aspire-pr-container/run-aspire-pr-container.sh "$@" } runner bash -lc 'curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- '"$prNumber" ``` On Windows PowerShell hosts, use the PowerShell runner instead: ```powershell $testDir = Join-Path $env:TEMP "aspire-pr-test-$([guid]::NewGuid().ToString('N'))" New-Item -ItemType Directory -Path $testDir -Force | Out-Null function runner { & ./eng/scripts/aspire-pr-container/run-aspire-pr-container.ps1 @args } $env:ASPIRE_PR_WORKSPACE = $testDir $env:ASPIRE_CONTAINER_USER = "0:0" runner bash -lc "curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- $prNumber" ``` For follow-up commands in the same mounted workspace, run: ```bash runner bash -lc '/workspace/.aspire/bin/aspire --version' ``` Because the container `HOME` is `/workspace`, the standard dogfood install still lands under `/workspace/.aspire`. The repo-local runner now backs `/workspace/.aspire` with a deterministic Docker-managed volume instead of the host bind mount, so follow-up commands can keep using `/workspace/.aspire/bin/aspire` and `/workspace/.aspire/hives/pr-<PR_NUMBER>/packages` without putting the AppHost RPC socket on the Docker Desktop workspace filesystem. To record the full host-side container session with asciinema, enable recording before invoking the runner. Recording is handled by the host-side runner script (not inside the container), so `asciinema` must be installed on the host. macOS/Linux/WSL example: ```bash export ASPIRE_PR_RECORD=1 export ASPIRE_PR_RECORDING_PATH="$testDir/pr-test.cast" # optional runner bash -lc 'curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- '"$prNumber" ``` Windows PowerShell example: ```powershell $env:ASPIRE_PR_RECORD = "1" $env:ASPIRE_PR_RECORDING_PATH = Join-Path $testDir "pr-test.cast" # optional runner bash -lc "curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- $prNumber" ``` #### Important template note When creating new projects from the PR build: - Prefer the downloaded PR hive explicitly instead of relying on channel resolution alone. - In non-interactive runs, pass both `--name` and `--output`. - For `aspire-starter`, also pass `--test-framework None --use-redis-cache false` unless the scenario explicitly needs those prompts. - Always pass `--localhost-tld false` to suppress the `Use *.dev.localhost URLs [y/N]:` prompt that causes `Failed to read input in non-interactive mode` errors. - Always pass `--suppress-agent-init` to suppress the post-create `Would you like to configure AI agent environments for this project?` prompt. - The `--output` directory must not already exist. `aspire new` refuses to write into a non-empty directory. If a previous attempt failed and left a partial directory behind, remove it before retrying (`Remove-Item -Recurse -Force` / `rm -rf`). - In TTY-attached runs where `--suppress-agent-init` is not passed, `aspire new` may ask `Would you like to configure AI agent environments for this project?`; answer explicitly (usually `n`) unless agent-init is part of the scenario. - Run `aspire new <template> --help` to discover all available flags for a template when you encounter unexpected prompts. New flags may be added between releases. Example starter-app automation (bash): ```bash projectName="PrSmoke" appRoot="$testDir/$projectName" "$cliPath" new aspire-starter \ --name "$projectName" \ --output "$appRoot" \ --source "$hivePath" \ --version "$cliVersion" \ --test-framework None \ --use-redis-cache false \ --localhost-tld false \ --suppress-agent-init ``` Example starter-app automation (PowerShell): ```powershell $projectName = "PrSmoke" $appRoot = "$testDir\$projectName" & $cliPath new aspire-starter ` --name $projectName ` --output $appRoot ` --source $hivePath ` --version $cliVersion ` --test-framework None ` --use-redis-cache false ` --localhost-tld false ` --suppress-agent-init ``` ### 4. Verify CLI Version Matches PR Commit Get the PR's head commit SHA and verify the installed CLI matches: ```bash # Get PR head commit SHA expectedCommit="$(gh pr view "$prNumber" --repo microsoft/aspire --json headRefOid --jq .headRefOid)" # Local mode: use the installed binary directly "$cliPath" --version # Container mode: use the installed binary in the mounted workspace runner bash -lc '/workspace/.aspire/bin/aspire --version' ``` **Important:** The installed binary path must be used for version checks (not bare `aspire`, which may resolve to some other install). The reported version should contain the PR head commit; matching the short SHA is sufficient. ### 5. Analyze PR Changes Examine the PR diff to understand what was changed: ```powershell # Get changed files gh pr view $prNumber --repo microsoft/aspire --json files --jq '.files[].path' # Get the PR diff gh pr diff $prNumber --repo microsoft/aspire ``` Categorize the changes: - **CLI changes**: Files in `src/Aspire.Cli/` - **Hosting changes**: Files in `src/Aspire.Hosting*/` - **Dashboard changes**: Files in `src/Aspire.Dashboard/` - **Client/Component changes**: Files in `src/Components/` - **Template changes**: Files in `src/Aspire.ProjectTemplates/` - **VS Code extension changes**: Files in `extension/` - **Test changes**: Files in `tests/` - **CI infrastructure changes**: GitHub Actions — files in `.github/workflows/`, `.github/actions/`, `.github/aw/`, `.github/agents/`, or CI scripts under `eng/scripts/`, `eng/github-ci/ci-skip-entirely-patterns.txt`, `eng/test-retry-patterns.json`; **and** Azure DevOps — `eng/pipelines/`, `eng/common/`, and the signing/packaging/publishing plumbing those pipelines call. More broadly, anything that changes *how CI selects, builds, or runs* (e.g. `tools/**` invoked by CI, MSBuild test plumbing in `eng/**/*.props`/`*.targets`, or CI config/data JSON), even when no `.github/` or `eng/pipelines/` file is touched. (Skill/doc-only edits under `.agents/skills/**` or `docs/**` are **not** tested by this skill — there is nothing to run for them.) > **If the PR touches CI infrastructure, follow the `ci-infra-testing.md` reference in this skill directory for that part.** Those changes are *not* validated by the CLI dogfood / template scenarios below. The reference has two tracks: **GitHub Actions** (most workflows don't run on the PR; unit tests don't catch trigger / permission / fork / portability / lock-drift gotchas) and **Azure DevOps** (no AzDO pipeline runs on a GitHub PR — a non-trivial change must be run on the `dnceng/internal` mirror via the `azdo-internal` skill). For an **infra-only** PR, skip the CLI install and template scenarios in Steps 3–9 entirely and use `ci-infra-testing.md` instead. For a mixed PR, do both. ### 6. Generate Test Scenarios Based on the PR changes, generate appropriate test scenarios. Always use new projects in the temp folder. Cover the expected happy path plus targeted unhappy-path, negative, and boundary cases that validate plausible misuse, invalid inputs, and failure states introduced or affected by the PR. #### Scenario Categories **For CLI changes (`src/Aspire.Cli/`):** - Test the specific command(s) that were modified - Run `aspire new` to verify basic functionality - Run `aspire run` to verify orchestration works - Test any new commands or options added **For Hosting integration changes (`src/Aspire.Hosting.*/`):** - Create a new Aspire project - Add the modified resource type to the AppHost - Run the application and verify the resource starts correctly - Check the Dashboard shows the resource properly **For Dashboard changes (`src/Aspire.Dashboard/`):** - Create and run an Aspire application - Navigate to the Dashboard - Take screenshots of relevant views - Verify the modified UI/functionality works **For Template changes (`src/Aspire.ProjectTemplates/`):** - Test creating projects from each modified template - Verify the generated project structure - Run the generated project **For Client/Component changes (`src/Components/`):** - Create a project that uses the modified component - Add the corresponding hosting resource - Test the client can connect to the resource **For CI infrastructure changes (`.github/**`, `eng/pipelines/**`, `eng/common/**`, CI scripts under `eng/`):**