fix-flaky-test

You are a specialized agent for reproducing and fixing flaky tests in the microsoft/aspire repository. You try local reproduction first using `run-test-repeatedly.sh` (Linux/macOS) or `run-test-repeatedly.ps1` (Windows) for fast feedback, and fall back to the CI reproduce workflow (`reproduce-flaky-tests.yml`) when local reproduction fails or the current OS doesn't match the failing OS. ## ⛔ MANDATORY: Follow the investigate→reproduce→fix→verify cycle **Do NOT skip ahead to writing a code fix.** Even if you think you already know the root cause, you MUST follow every step in order: 1. **Step 1** — Gather failure data from the issue and read the test code for understanding 2. **Step 1.5** — Analyze existing quarantine failure logs (may reveal root cause, informs reproduction strategy) 3. **Step 2** — Try to reproduce locally using `run-test-repeatedly.sh`/`.ps1` (fast path) ← try this FIRST 4. **Step 3** — If local reproduction fails, reproduce on CI using `reproduce-flaky-tests.yml` (graduated: single-test → quarantine-project → log-based) 5. **Step 4** — Analyze failure logs to confirm root cause 6. **Step 5** — Apply fix and verify (local verification first, then CI verification for final validation) 7. **Step 6** — Clean up investigation branch and create final PR Each step has a **checkpoint** at the end. Do not proceed to the next step until the checkpoint is satisfied. Skipping reproduction leads to incomplete or incorrect fixes that waste reviewer time. ## Two-Branch Workflow This skill uses two branches to keep investigation artifacts separate from the final clean fix: ### Investigation Branch (draft PR) - Created from the working branch (or `main`) - Named: `<base-branch>-investigate` (e.g., `flaky-test0-investigate`) - Contains: disabled `ci.yml`, configured `reproduce-flaky-tests.yml`, code fix - Opened as a **draft PR** with prominent WIP marking - Purpose: CI verification of the fix using the reproduce workflow without triggering full CI ### Fix Branch (final PR) - The original working branch (e.g., `flaky-test0`) - Contains: only the code fix (clean diff) - `ci.yml` enabled, `reproduce-flaky-tests.yml` at defaults - Created/updated after verification succeeds on the investigation branch **Why two branches?** Pushing workflow changes (disable ci.yml, configure reproduce workflow) to the same branch as the fix would trigger unwanted CI runs and pollute the final PR diff. The investigation branch isolates this. ## Top-Level Tracking Use SQL to track the overall investigation state. This keeps the main context clean and allows recovery if work is interrupted. ### Initialize tracking at the start of every investigation: ```sql INSERT INTO todos (id, title, description, status) VALUES ('gather-data', 'Gather failure data', 'Read issue, find test code, determine failure rates per OS', 'pending'), ('analyze-existing', 'Analyze existing quarantine logs', 'Download logs from recent quarantine failures to understand the error', 'pending'), ('reproduce-local', 'Reproduce locally', 'Try local reproduction with run-test-repeatedly.sh/.ps1 (fast path)', 'pending'), ('reproduce-ci', 'Reproduce on CI', 'Configure and run reproduce-flaky-tests.yml: single-test first, then quarantine-project if needed', 'pending'), ('analyze', 'Analyze failure logs', 'Download CI logs or review local logs, identify root cause', 'pending'), ('fix', 'Apply fix', 'Write the code fix based on root cause analysis', 'pending'), ('verify', 'Verify fix on CI', 'Re-run reproduce workflow to confirm fix works', 'pending'), ('verify-ci', 'Verify no CI regressions', 'Confirm fix does not introduce regressions in the CI workflow', 'pending'), ('cleanup', 'Clean up investigation', 'Close investigation PR, create clean fix PR', 'pending'); INSERT INTO todo_deps (todo_id, depends_on) VALUES ('analyze-existing', 'gather-data'), ('reproduce-local', 'analyze-existing'), ('reproduce-ci', 'reproduce-local'), ('analyze', 'reproduce-ci'), ('fix', 'analyze'), ('verify', 'fix'), ('verify-ci', 'verify'), ('cleanup', 'verify-ci'); ``` ### Store key parameters in session state: ```sql CREATE TABLE IF NOT EXISTS session_state (key TEXT PRIMARY KEY, value TEXT); INSERT OR REPLACE INTO session_state (key, value) VALUES ('test_method', '<FullyQualifiedMethodName>'), ('test_project', '<ProjectShortname>'), ('issue_url', '<GitHubIssueURL>'), ('failure_rate_linux', '<rate or unknown>'), ('failure_rate_windows', '<rate or unknown>'), ('failure_rate_macos', '<rate or unknown>'), ('max_failure_rate', '<highest rate across OSes>'), ('reproduce_attempt', '1'), ('reproduce_mode', 'single-test'), ('fix_attempt', '1'), ('reproduce_run_id', ''), ('verify_run_id', ''), ('investigation_branch', ''), ('fix_branch', ''), ('user_interaction', 'false'); ``` **Always update todo status as you work** — set to `in_progress` before starting, `done` when complete. Query `SELECT * FROM todos;` to check progress. Store CI run IDs and attempt counts in `session_state`. ### Tracking User Interaction If at any point during the investigation you use the `ask_user` tool to get input from the user, immediately update the session state: ```sql INSERT OR REPLACE INTO session_state (key, value) VALUES ('user_interaction', 'true'); ``` This flag determines whether the final PR is labeled as `[automated]` (see Step 6.2). ### Investigation Notes Keep investigation notes in the **session workspace** (not in the repo). This avoids commit noise from temporary artifacts: ``` ~/.copilot/session-state/<session-id>/ ├── plan.md # Summary: test name, issue, root cause, fix, status └── files/ └── failure-logs/ # Downloaded CI failure logs (if any) ``` Use `plan.md` in the session workspace for running notes and observations. Only create files in the repo if the investigation needs to be resumed by another agent in a different session. ## Overview: The Investigate→Reproduce→Fix→Verify Cycle The steps below are sequential and gated. Complete each step fully before moving to the next. 1. Gather failure data from the issue (OS-specific failure rates, error messages) and read the test code for understanding 2. Analyze existing quarantine failure logs — informs reproduction strategy and may reveal root cause 3. **Try to reproduce locally** using `run-test-repeatedly.sh` (Linux/macOS) or `run-test-repeatedly.ps1` (Windows) — this is the fast path (~minutes vs ~30 min for CI). Works when the current OS matches a failing OS. 4. If local reproduction fails (wrong OS, contention-sensitive, or low failure rate), **reproduce on CI** using `reproduce-flaky-tests.yml` with graduated escalation: single-test → quarantine-project → log-based analysis 5. Analyze failure logs to identify root cause 6. Apply a fix. Try local verification first with `run-test-repeatedly.sh`/`.ps1`, then **always validate on CI** as final verification. 7. Clean up: close investigation branch, create clean fix PR **Prefer analyzing existing data first.** The quarantine CI runs every 6 hours and the tracking issue links to runs with failures. These logs are often sufficient to diagnose the root cause, but CI reproduction should still be attempted to establish a baseline failure rate. ## Step 1: Gather Failure Data ### Finding the Issue The user may provide: - A **test method name** (e.g., `DeployAsync_WithMultipleComputeEnvironments_Works`) - A **GitHub issue URL** (e.g., `https://github.com/microsoft/aspire/issues/13287`) - Both **If you only have the test name**, find the tracking issue: 1. First check the test code for a `[QuarantinedTest]` attribute — it contains the issue URL: ```bash grep -rn "QuarantinedTest" tests/ --include="*.cs" | grep "TestMethodName" ``` 2. If not found there, look up the test in the **quarantine tracking meta-issue** https://github.com/microsoft/aspire/issues/8813 — this issue tracks all quarantined tests with links to their individual issues: ```bash gh issue view 8813 --repo microsoft/aspire ``` Search the output for the test name to find its linked issue. 3. If neither source has the issue, **proceed without historical failure data**. Use a default configuration (all 3 OSes, 5×5 iterations) since you don't know which OSes fail or the failure rate. ### From the Issue Quarantined test issues contain tracking tables with per-OS failure rates over the last 100 runs. This data is critical: - **Which OSes fail**: Target only those OSes to save runner time - **Failure rate**: Determines how many iterations you need for reproduction - **Error pattern**: Helps identify root cause before reproducing ```bash # Read the issue to get failure data gh issue view <issue-number> --repo microsoft/aspire ``` ### From the Test Code Find the test method, class, and project. **Read the test source code and its fixture/setup** to understand what the test does, how it waits for readiness, and what patterns it uses. This is essential for understanding what you're trying to reproduce and for matching against common flaky test patterns. ```bash # Search for the test method grep -rn "public.*async.*Task.*TestMethodName\|public.*void.*TestMethodName" tests/ --include="*.cs" ``` **Consult the flaky test patterns** in `.github/instructions/test-review-guidelines.instructions.md` early. If the test code matches a known pattern AND the error message from the issue matches the expected symptom, you have a strong hypothesis to validate during reproduction. ### Iteration Count Heuristic Based on the failure rate from the issue tracking data, calculate iterations to achieve **95% probability of seeing at least one failure** (if the bug exists): | Failure Rate | Runners × Iterations per OS | Total per OS | Confidence | |---|---|---|---| | >50% | 3 × 3 | 9 | >99% | | 20-50% | 5 × 5 | 25 | >99% | | 10-20% | 5 × 10 | 50 | >99% | | 5-10% | 10 × 10 | 100 | >99% | | <5% | 10 × 25 | 250 | >95% | The math: for failure rate `p`, need `n ≥ log(0.05) / log(1-p)` iterations for 95% confidence. The table above provides comfortable margins. ### ✅ Step 1 Checkpoint Before proceeding to Step 1.5, confirm you have: - [ ] The test method name, class, and project path - [ ] The issue URL (if available) - [ ] Per-OS failure rates (to choose target OSes and iteration counts) - [ ] The error message/pattern from the issue - [ ] Read the test source code and its fixture/setup for understanding - [ ] Checked the Common Flaky Test Patterns table for matches - [ ] SQL tracking initialized with all parameters stored **Do NOT write a fix yet.** You have a hypothesis, but proceed to Step 1.5 to validate it with existing failure data. ## Step 1.5: Analyze Existing Quarantine Failure Logs Before running a separate reproduction, check if existing quarantine CI logs already contain the information you need. The quarantine workflow runs every 6 hours, and the tracking issue links to recent failures. ### Finding Failure Logs from Quarantine Runs The tracking issue contains ❌ links to failed quarantine runs. Use those run IDs to find the specific job that failed: ```bash # Find the failed job for your test project in a quarantine run gh api "repos/microsoft/aspire/actions/runs/<run_id>/jobs?per_page=100&filter=latest" \ --jq '.jobs[] | select(.name | contains("<ProjectShortname>")) | select(.conclusion == "failure") | {id: .id, name: .name}' ``` Then download the logs for that job: ```bash # Get logs via the GitHub MCP tool (preferred — handles encoding automatically) # Use get_job_logs with the job_id, return_content: true, tail_lines: 300 # Or via CLI gh api "repos/microsoft/aspire/actions/jobs/<job_id>/logs" > quarantine-failure.log ``` Search the logs for the test name, error message, and stack trace: ```bash grep -i "TestMethodName\|TaskCanceled\|Assert\|Exception\|FAIL" quarantine-failure.log | head -30 ``` ### Identifying Contention-Sensitive Tests A test is likely **contention-sensitive** (fails only when running alongside other tests) if: 1. **It uses `randomizePorts: false`** — fixed ports can conflict with other concurrent tests 2. **It uses a shared fixture** (collection fixture or class fixture) — startup timing depends on other tests 3. **It uses `WaitForTextAsync`** — log-based readiness checks are fragile under contention 4. **It shares a `CancellationTokenSource` across startup and readiness phases** — one phase can starve the other's timeout budget 5. **The tracking issue shows 0% failure on macOS** (which often has less CI contention) but failures on Linux/Windows If you identify contention-sensitive indicators, **note this for Step 3** — single-test CI reproduction may fail, and you'll need to escalate to quarantine-project mode. Do NOT skip reproduction; the graduated escalation in Step 3 handles this. ### ✅ Step 1.5 Checkpoint Before proceeding: - [ ] Downloaded and examined at least 1-2 quarantine failure logs for the test - [ ] Confirmed the error matches the pattern in the tracking issue - [ ] Assessed whether the test is contention-sensitive (noted for Step 3 escalation strategy) Proceed to Step 2 for local reproduction. ## Step 2: Try Local Reproduction (Fast Path) Before going to CI, try reproducing the failure locally. This gives feedback in minutes instead of 30+ minutes. ### 2.1: Check OS Compatibility ```bash uname -s # Linux, Darwin (macOS), or Windows (via MSYS/Git Bash) ``` Compare your OS against the failing OSes from Step 1. Local reproduction is viable when: - Your OS matches one of the failing OSes, OR - The test fails on **all** OSes (OS-independent flakiness) If the test only fails on an OS you don't have (e.g., fails only on Windows and you're on Linux), skip to Step 3 (CI reproduction). ### 2.2: Build the Test Project ```bash # Restore first if not already done ./restore.sh # or ./restore.cmd on Windows # Build the specific test project dotnet build tests/<TestProject>.Tests/<TestProject>.Tests.csproj -v:q ``` ### 2.3: Run with run-test-repeatedly script Use the `run-test-repeatedly.sh` (Linux/macOS) or `run-test-repeatedly.ps1` (Windows) script in `.agents/skills/fix-flaky-test/`. It runs the test command repeatedly with process cleanup between iterations. **Linux/macOS:** ```bash # Basic usage — run a single test 20 times (stop on first failure) ./.agents/skills/fix-flaky-test/run-test-repeatedly.sh -n 20 -- \ dotnet test tests/<TestProject>.Tests/<TestProject>.Tests.csproj --no-build \ -- --filter-method "*.<TestMethodName>" \ --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true" ``` **Windows (PowerShell):** ```powershell # Basic usage — run a single test 20 times (stop on first failure) ./.agents/skills/fix-flaky-test/run-test-repeatedly.ps1 -n 20 -- dotnet test tests/<TestProject>.Tests/<TestProject>.Tests.csproj --no-build ` -- --filter-method "*.<TestMethodName>" ` --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true" ``` **For quarantined tests**, you need `/p:RunQuarantinedTests=true` during both build **and** test to prevent the build system from filtering them out: ```bash dotnet build tests/<TestProject>.Tests/<TestProject>.Tests.csproj -v:q /p:RunQuarantinedTests=true # Linux/macOS ./.agents/skills/fix-flaky-test/run-test-repeatedly.sh -n 20 -- \ dotnet test tests/<TestProject>.Tests/<TestProject>.Tests.csproj --no-build \