macios-ci-postmortem

# macios CI Post-Mortem Analyze CI failures across recent PRs to identify flaky tests, infrastructure issues, and shared regressions that are not caused by any specific PR. File or update GitHub issues for these. ## References Read these as needed during investigation: - `references/azure-devops-cli.md` — az CLI commands, artifact naming conventions, and JSON parsing caveats. ## Overview This skill operates in four phases: 1. **Discovery** — collect all recent PR-validation builds from AzDO 2. **Extraction** — for failed builds, extract normalized failure records 3. **Classification** — categorize failures as flaky, infrastructure, shared regression, or PR-specific 4. **Issue Actions** — propose GitHub issues, get user confirmation, then file/update ## Phase 1: Discovery — Collect Recent Builds **Start from builds, not PRs.** This is faster, gives access to commit SHAs for rerun detection, and captures builds for PRs that may already be closed. ### Step 1.1: List recent PR-validation builds Use the `az` CLI to get builds from the last 7 days. The macios CI runs on `devdiv.visualstudio.com/DevDiv`. ```bash # Get the date 7 days ago in ISO format SINCE=$(python3 -c "from datetime import datetime, timedelta; print((datetime.utcnow() - timedelta(days=7)).strftime('%Y-%m-%dT%H:%M:%SZ'))") # List recent builds for the PR pipeline az pipelines build list \ --org https://devdiv.visualstudio.com \ --project DevDiv \ --reason pullRequest \ --result failed \ --top 200 \ -o json > /tmp/postmortem_builds.json ``` Also fetch partially succeeded builds (these contain test failures): ```bash az pipelines build list \ --org https://devdiv.visualstudio.com \ --project DevDiv \ --reason pullRequest \ --result partiallySucceeded \ --top 200 \ -o json > /tmp/postmortem_builds_partial.json ``` ### Step 1.2: Parse and filter builds ```python import json from datetime import datetime, timedelta, timezone since = datetime.now(timezone.utc) - timedelta(days=7) def load_builds(path): with open(path) as f: content = f.read() return json.JSONDecoder().raw_decode(content)[0] builds = load_builds('/tmp/postmortem_builds.json') + load_builds('/tmp/postmortem_builds_partial.json') # Filter to last 7 days and macios pipelines recent = [] for b in builds: finish = b.get('finishTime', '') if not finish: continue ft = datetime.fromisoformat(finish.replace('Z', '+00:00')) if ft < since: continue # Only include macios pipelines defn = b.get('definition', {}).get('name', '') if 'macios' not in defn.lower() and 'xamarin-macios' not in defn.lower(): continue recent.append({ 'id': b['id'], 'result': b['result'], 'pr': b.get('triggerInfo', {}).get('pr.number', ''), 'sourceBranch': b.get('sourceBranch', ''), 'sourceVersion': b.get('sourceVersion', ''), # commit SHA — critical for rerun detection 'pipeline': defn, 'finishTime': finish, }) print(f"Found {len(recent)} builds from {len(set(b['pr'] for b in recent if b['pr']))} PRs") ``` ### Step 1.3: Group builds for rerun detection Group by `(pr, pipeline, sourceVersion)`. Multiple builds with the same commit SHA for the same PR/pipeline are reruns. ```python from collections import defaultdict # Group: (pr, pipeline, commitSHA) -> [builds] groups = defaultdict(list) for b in recent: key = (b['pr'], b['pipeline'], b['sourceVersion']) groups[key].append(b) # Also group by just (pr, pipeline) to see if new commits fixed things pr_pipeline = defaultdict(list) for b in recent: key = (b['pr'], b['pipeline']) pr_pipeline[key].append(b) ``` ## Phase 2: Extraction — Get Failure Details For each failed/partiallySucceeded build, extract failure information. Use a SQL database to track failures across builds. ### Step 2.1: Set up failure tracking ```sql CREATE TABLE IF NOT EXISTS ci_failures ( id INTEGER PRIMARY KEY AUTOINCREMENT, build_id INTEGER, pr TEXT, pipeline TEXT, commit_sha TEXT, finish_time TEXT, job_name TEXT, failure_type TEXT, -- 'TestFailure', 'BuildFailure', 'TimedOut', 'Crashed', 'Infrastructure' test_fullname TEXT, -- e.g. 'MonoTouchFixtures.SomeTest.TestMethod' platform TEXT, -- e.g. 'ios', 'tvos', 'macos', 'maccatalyst' config TEXT, -- e.g. 'Debug (ARM64)', 'Release (x64)' error_signature TEXT, -- normalized error message / top stack frame raw_message TEXT ); ``` ### Step 2.2: For each build, get the timeline and TestSummary artifacts Only process builds with failures. For efficiency, first check the timeline for failed jobs, then only download artifacts for those jobs. ```bash # Get timeline az devops invoke --area build --resource timeline \ --route-parameters project=DevDiv buildId=<buildId> \ --org https://devdiv.visualstudio.com -o json > /tmp/timeline_<buildId>.json ``` Parse the timeline to find failed jobs: ```python import json with open(f'/tmp/timeline_{build_id}.json') as f: data = json.JSONDecoder().raw_decode(f.read())[0] failed_jobs = [] for r in data.get('records', []): if r.get('type') == 'Job' and r.get('result') == 'failed': failed_jobs.append({ 'name': r['name'], 'id': r['id'], 'logId': r.get('log', {}).get('id'), }) ``` ### Step 2.3: Download TestSummary artifacts (fast triage) TestSummary artifacts are small and quick to download. Use them first to identify which jobs failed: ```bash artifact="TestSummary-simulator_tests<jobname>-1" mkdir -p "/tmp/postmortem/${build_id}/${artifact}" az pipelines runs artifact download \ --artifact-name "$artifact" \ --path "/tmp/postmortem/${build_id}/${artifact}" \ --run-id <buildId> \ --org https://devdiv.visualstudio.com --project DevDiv ``` Parse the TestSummary.md to determine which jobs have test failures. This is the first-pass filter. ### Step 2.4: Download HtmlReport artifacts (deep analysis) **This is the most time-consuming step — minimize downloads aggressively.** Each test run produces an HtmlReport artifact (60-140MB zip) containing: - `tests/index.html` — Main report with all test configurations, pass/fail, inline failure details - `tests/<suite>/<num>/test-<platform>-<timestamp>.xml` — NUnit XML with individual test-case results - `tests/<suite>/<num>/results-<timestamp>.xml` — NUnit results for dotnettests **CRITICAL: Only download HtmlReport zips for jobs where TestSummary shows TEST failures (❌ markers).** Do NOT download HtmlReports for: - Build failures (no test results exist — the build didn't get far enough to run tests) - Infrastructure failures (bot provisioning, timeout, etc.) - Jobs where TestSummary shows all tests passed (the job may have failed for other reasons) To find exact artifact names, first list artifacts for the build: ```bash az pipelines runs artifact list --run-id <buildId> \ --org https://devdiv.visualstudio.com --project DevDiv -o json ``` Then download only matching HtmlReport artifacts for jobs with test failures: ```bash az pipelines runs artifact download \ --artifact-name "HtmlReport-<exact-name>-1" \ --path "/tmp/postmortem_deep/" \ --run-id <buildId> \ --org https://devdiv.visualstudio.com --project DevDiv ``` **Performance note:** Each download takes 1-3 minutes (sequential, no parallelism in az CLI). Downloading 500 artifacts takes ~2 hours. By filtering with TestSummary first, you can typically reduce this to 50-100 artifacts. ### Step 2.5: Parse NUnit XML for individual test failures Extract individual test failures from the NUnit XML files inside the HtmlReport zips: ```python import zipfile, xml.etree.ElementTree as ET, html def extract_failures_from_nunit_xml(xml_content): """Parse NUnit XML to extract individual failing test cases.""" root = ET.fromstring(xml_content) failures = [] for tc in root.iter('test-case'): if tc.get('result') == 'Failed': name = tc.get('fullname', 'Unknown') msg_el = tc.find('.//failure/message') stack_el = tc.find('.//failure/stack-trace') failures.append({ 'test': name, 'message': msg_el.text if msg_el is not None else '', 'stack': stack_el.text[:500] if stack_el is not None else '', }) return failures # Process a zip file with zipfile.ZipFile('/tmp/postmortem_deep/html_BUILDID_JOB.zip') as zf: for name in zf.namelist(): if name.endswith('.xml') and 'test-' in name and '-clean' not in name: xml_content = zf.read(name).decode('utf-8', errors='replace') failures = extract_failures_from_nunit_xml(xml_content) ``` **Important**: Skip files ending in `-clean.xml` (these are filtered versions). The root XML tag is `TouchUnitTestRun` (not standard NUnit format, but `test-case` elements follow standard structure). For **dotnettests**, individual test failures are listed inline in `<li>` tags in the HTML (not in separate XML). Parse these from `tests/index.html`: ```python import re # Pattern for inline test failures in dotnettests HTML failures_in_html = re.findall(r'<li[^>]*>([^<]*(?:Failed|Error)[^<]*)</li>', html_content) ``` ### Step 2.6: Handle crashes and build failures When a test runner crashes (exit code 134, etc.) or a build fails before tests run, there will be **no NUnit XML results**. These appear in the HTML as: - `Test run crashed (exit code: NNN)` - `BuildFailure` Capture these from the HTML and record them as separate failure types (CRASH, BUILD_FAILURE). ### Step 2.6a: Collect detailed information for build failures For any failure that involves a **build error** (a test suite that fails to build, or a unit test that builds something and the build fails), collect as much detail as possible: #### 1. Extract specific build error messages When the NUnit failure message says something like `'dotnet build' failed with exit code 1`, that alone is not useful. Look for the actual compiler/linker/MSBuild errors in: - The NUnit failure `message` and `stack-trace` elements (sometimes the full build output is captured there) - The HtmlReport `index.html` — build errors are often shown inline - The build step log (download via `az devops invoke --area build --resource logs`) Look for patterns like: - `error CS####:` (C# compiler errors) - `error MT####:` / `error MM####:` (mtouch/mmp errors) - `error MSB####:` (MSBuild errors) - `error IL####:` (ILLink/trimmer errors) - `error NETSDK####:` (SDK errors) #### 2. Collect binlogs when available Binlog files (`.binlog`) contain the full MSBuild log and are invaluable for diagnosing build failures. They are often available as build artifacts. ```bash # List artifacts to find binlog-related ones az pipelines runs artifact list --run-id <buildId> \ --org https://devdiv.visualstudio.com --project DevDiv -o json \ | python3 -c "import json,sys; [print(a['name']) for a in json.load(sys.stdin) if 'binlog' in a['name'].lower() or 'Binlog' in a['name']]" ``` If binlogs are inside the HtmlReport zip (common path: `tests/<suite>/<num>/*.binlog` or referenced in test output), extract them. Binlogs may also be embedded in test failure messages or stack traces as file paths — note these paths for reference. #### 3. Attach binlogs to issues When filing an issue for a build failure: - Download the relevant binlog file - Zip it (GitHub doesn't allow `.binlog` attachments, but `.zip` is fine) - Attach the zipped binlog to the issue using `gh issue comment` with the `--attach` flag, or by uploading via the GitHub API ```bash # Download a binlog artifact az pipelines runs artifact download \ --artifact-name "<binlog-artifact-name>" \ --path "/tmp/postmortem_binlogs/" \ --run-id <buildId> \ --org https://devdiv.visualstudio.com --project DevDiv # Zip it for attachment zip /tmp/postmortem_binlogs/build_<buildId>.binlog.zip /tmp/postmortem_binlogs/*.binlog # Attach to issue (if the gh CLI version supports --attach, otherwise note the link) ``` #### 4. Include build errors in the issue body Always include the specific build error messages in the issue body. Example: ```markdown ### Build Errors The `dotnet build` step failed with the following errors: ``` error CS8602: Dereference of a possibly null reference. [src/Foo/Bar.csproj] error MT0099: No platform assembly! [src/Baz/Qux.csproj] ``` **Binlog:** [build_14017033.binlog.zip](link-to-attachment) (attached) ``` This makes the issue actionable without requiring the reader to navigate through AzDO build logs. ### Step 2.7: For infrastructure/setup failures without TestSummary Check the timeline for failed tasks in setup/provisioning stages. Extract error info from task log lines: ```bash az devops invoke --area build --resource logs \ --route-parameters project=DevDiv buildId=<buildId> logId=<logId> \ --org https://devdiv.visualstudio.com -o json > /tmp/log_<buildId>_<logId>.json ```