ci-status

# CI Status Skill Provide a dashboard view of SkiaSharp CI health across main and recent release branches, with AI-powered analysis to identify patterns, regressions, and actionable fixes. This skill gives a **broad overview** of CI health across multiple branches, including the current dnceng Build and Tests pipelines. ## Pipelines Tracked ### Prerequisites The collector script requires: - **`az` CLI** — authenticated with access to `dnceng-public/public` and `dnceng/internal` - **`gh` CLI** — authenticated with read access to `mono/SkiaSharp` and `mono/SkiaSharp-API-docs` - **Git remotes** — fetched recently so `git branch -r` returns up-to-date release branches ### Public CI (dnceng-public/public org — triggers on push/PR to main and release/*) | Pipeline Name | Org/Project | Definition ID | URL | |---------------|-------------|---------------|-----| | `mono-SkiaSharp` | dnceng-public/public | 345 | [link](https://dev.azure.com/dnceng-public/public/_build?definitionId=345&_a=summary) | ### Internal Release Chain (dnceng/internal — triggers on release/* branches) | Order | Pipeline Name | Definition ID | URL | |-------|---------------|---------------|-----| | 1 | `skiasharp-package` | 1642 | [link](https://dev.azure.com/dnceng/internal/_build?definitionId=1642) | | 2 | `skiasharp-tests` | 1630 | [link](https://dev.azure.com/dnceng/internal/_build?definitionId=1630) | Tests consumes the folder-qualified pipeline resource `\dotnet\skiasharp\skiasharp-package`. The Build pipeline owns native and managed compilation, real signing, BAR registration/validation, and Arcade's standard Darc/Maestro stages. ### GitHub Actions (SkiaSharp and stable API-docs publication) | Workflow | Repository | Trigger | Why Track | |----------|------------|---------|-----------| | Pages - Deploy | mono/SkiaSharp | Push/PR to main | Docs site broken if failing | | Pages - Go Live! | mono/SkiaSharp | Workflow dispatch | Docs don't publish if failing | | Pages - PR Staging - Cleanup | mono/SkiaSharp | PR close events | Stale staging deploys accumulate | | Pages - PR Staging - Sweep Stale | mono/SkiaSharp | Daily (06:00 UTC) | Stale staging deploys accumulate | | Sync - Samples | mono/SkiaSharp | Push/PR to `samples/` | Sample projects broken if failing | | Tests - Binding Generation Determinism | mono/SkiaSharp | Push/PR | Generated bindings drift undetected | | Sync - Skia Submodule | mono/SkiaSharp | Daily (10:30 UTC) | Skia submodule pin goes stale | | Sync - Release Notes & API Diffs | mono/SkiaSharp | Push to main + daily | Release notes stop auto-updating | | Sync - Skia Upstream | mono/SkiaSharp | Every 6h | Upstream tracking breaks | | Fixer - Memory Leak | mono/SkiaSharp | Every 12h | Leak-hunting automation stops | | Fixer - Performance | mono/SkiaSharp | Every 12h | Perf-hunting automation stops | | Sync - Issue Triage | mono/SkiaSharp | Daily | Triage automation stops | | Sync - Issue Template Versions | mono/SkiaSharp | Daily (09:00 UTC) | Bug template advertises stale versions | | Sync - Agentic Data | mono/SkiaSharp | Workflow run events | AI workflow data lost | | Track - Artifact Sizes | mono/SkiaSharp | Nightly + PR sweep | Package size tracking/PR reports stop | | Track - Benchmarks | mono/SkiaSharp | Every 6h | Benchmark trend tracking stops | | Release - Prepare | mono/SkiaSharp | Workflow dispatch | Release branches cannot be cut | | Release - Finish | mono/SkiaSharp | Workflow dispatch | Releases cannot be finalized | | Release - Milestones | mono/SkiaSharp | Workflow dispatch | Milestone reconciliation stops | | Update GitHub Release summaries | mono/SkiaSharp | Workflow dispatch | Release bodies go stale | | Release - Tooling Tests | mono/SkiaSharp | Push/PR to release tooling | Release scripts regress unnoticed | | Automation - Tooling Tests | mono/SkiaSharp | Push/PR to automation tooling | Automation scripts regress unnoticed | | PR - Backport | mono/SkiaSharp | PR label/comment | Cherry-picks to release branches fail | | PR - Rebase | mono/SkiaSharp | PR comment | PR rebase automation broken | | PR - Artifacts Comment | mono/SkiaSharp | Workflow run events | Build links not posted to PRs | | Merge Message | mono/SkiaSharp | PR comment events | Merge commit messages not drafted | | Go Live | mono/SkiaSharp-API-docs | Workflow dispatch | API reference docs do not publish to Microsoft Learn | > Schedules above are deliberately imprecise for gh-aw generated `*.lock.yml` workflows > ("Every 6h", "Daily"). The compiler re-jitters their cron on every upgrade, so a literal > `HH:MM UTC` here would silently go stale. `scripts/tests/test_workflow_registry.py` > enforces this: any time it *does* find documented must match the workflow file. > External API documentation regeneration is intentionally not tracked. Its workflow in > `mono/SkiaSharp-API-docs` is maintained independently and its name is currently in flux; > add it to the registry only after a stable workflow name is established. --- ## Step 1: Collect Raw Data Run the collector script to gather build status, issues, and commit info: ```bash python3 .agents/skills/ci-status/scripts/ci-status.py \ --json output/ai/ci-status-data.json ``` This produces `ci-status-data.json` containing raw pipeline runs, GitHub Actions statuses, regression markers, issues/errors, and associated commits. The script also prints a console summary for quick visual inspection. ### Options | Flag | Default | Description | |------|---------|-------------| | `--branches N` | 3 | Number of most recent release/* branches to include | | `--builds N` | 5 | Number of recent builds to show per pipeline per branch | | `--no-issues` | off | Skip fetching errors/warnings (faster, less detail) | | `--json PATH` | none | Write raw structured JSON (for AI analysis) | ### Examples ```bash # Standard collection python3 .agents/skills/ci-status/scripts/ci-status.py --json output/ai/ci-status-data.json # Quick check (no timeline fetch) python3 .agents/skills/ci-status/scripts/ci-status.py --no-issues # Deep analysis window (10 builds, 5 branches) python3 .agents/skills/ci-status/scripts/ci-status.py --branches 5 --builds 10 --json output/ai/ci-status-data.json ``` ### File Boundary | File | Produced By | Consumed By | |------|-------------|-------------| | `output/ai/ci-status-data.json` | Collector script (Step 1) | AI analysis only (Step 2) | | `output/ai/ci-status-YYYY-MM-DD.json` | AI (Step 3) | Validator + Renderers (Steps 4–5) | > ⚠️ **Never pass `ci-status-data.json` to validate or render scripts.** Those scripts expect the > augmented report JSON that the AI assembles in Step 3. --- ## Step 2: AI Analysis After the script runs, read the JSON data (`output/ai/ci-status-data.json`) and perform the following analysis. **All claims must reference actual build IDs and URLs from the data.** ### 2.1 Executive Summary (Verdict) Classify overall health as one of: - 🟢 **Healthy** — all branches green or only warnings (both AzDO and GitHub Actions) - 🟡 **Degraded** — some failures but main is green - 🔴 **Broken** — main is red or a release branch is blocked Write 1-2 sentences: what's broken, since when, and the top action. ### 2.2 Root Cause Clustering Group all errors/warnings across all branches and pipelines by **normalized signature**: 1. Strip volatile fragments (build IDs, timestamps, GUIDs, file paths with random hashes, agent names) 2. Cluster on `(task_name, normalized_first_error_line)` 3. For each cluster, determine: - **Category**: one of `code_regression`, `flake`, `infra_network`, `quota_resource`, `chain_blockage`, `unknown` - **Footprint**: which branches × pipelines are affected - **First/last seen** within the window - **Sample error** (verbatim from data) > ⚠️ **Important:** Finalize root-cause clusters only AFTER performing the Pipeline Chain Analysis (§2.3). Downstream cascade failures must be collapsed into the upstream root cause, not counted as independent clusters. #### Classification Decision Tree | Signal | Category | |--------|----------| | Message contains `network`, `timeout`, `EOF`, `connection`, `nuget.org`, `429`, `503` | infra_network | | Message contains `No space left`, `OOM`, `killed`, `agent lost`, `pool` | quota_resource | | Same build passes/fails with no code change (pass/fail/pass pattern) | flake | | Failure appears on multiple unrelated branches simultaneously | infra_network | | Failure appears at a green→red transition on one branch only | code_regression | | Downstream pipeline failed but upstream in same chain also failed | chain_blockage | | None of the above | unknown | ### 2.3 Pipeline Chain Analysis (MANDATORY — always perform, even when failures look independent) The Internal chain is sequential: **Build -> Tests**. A red Tests run is not automatically an independent failure; it may be a downstream casualty of the exact Build resource it consumed. For **every** branch that has ≥1 red internal pipeline, you MUST: 1. Identify whether the connected Build failed before Tests. 2. Decide whether a red Tests run is an **independent failure** (its own distinct error) or a **cascade** from the exact Build resource. 3. **Collapse cascades** into the Build root cause so a single break is not counted twice. Emit one explicit sentence per affected branch, even if the answer is "no cascade": - Cascade: `"release/X: Build and Tests are red; Tests consumed that exact Build and is a downstream cascade, not an independent break."` - Independent: `"release/X: Build and Tests are both red but have unrelated errors ({errA} vs {errB}) — two independent failures."` ⚠️ Do not skip this step or list internal pipelines as separate equal-weight failures without first stating the chain verdict. This is the most common analysis miss. ### 2.4 Regression Detection For each branch × pipeline, find green→red transitions (the script pre-computes these in `regression` fields): - Report the last green build and the first red build - List the associated commits from `changes` — these are the regression suspects - Provide link to the first red build for investigation ### 2.5 Flake Detection Look for alternating pass/fail patterns within a single branch × pipeline: - ✅ ❌ ✅ or ❌ ✅ ❌ pattern = likely flake - Same error appearing intermittently (present in some builds, absent in others with no code change) ### 2.6 Cross-Branch Correlation | Pattern | Interpretation | |---------|---------------| | Same error on ≥2 unrelated branches | Infrastructure issue (toolchain, agent pool, NuGet) | | Error on exactly one branch | Code regression specific to that branch | | Same error on release/* and main | Shared code issue (or infra) | | Error only on release/X.Y.x but not release/X.Y.Z | Servicing-specific backport issue | ### 2.7 Release Risk Assessment For each `release/*` branch: - Is it shippable right now? (yes / no / blocked) - Days since last full green chain - Blockers (link to root cause clusters) - Recommendation: `ship`, `wait`, `cherry-pick`, `investigate` ### 2.8 GitHub Actions Health For each tracked GitHub Actions workflow: - Current status (pass/fail/skipped/no recent runs) - Any failures or patterns (recurring failures, recent regressions) - Whether failures are related to AzDO failures (same commit?) or independent - Categorize by severity: - **High**: Pages - Deploy, Sync - Samples, Sync - Release Notes & API Diffs, Sync - Skia Upstream, Release - Prepare, Release - Finish, Release - Tooling Tests (broken = user-facing impact or release process blocked) - **Medium**: Go Live, Sync - Skia Submodule, Fixer - Memory Leak, Fixer - Performance, Sync - Issue Triage, Sync - Issue Template Versions, Tests - Binding Generation Determinism, Automation - Tooling Tests, Release - Milestones, Update GitHub Release summaries, PR - Backport, Pages - Go Live! (broken = automation degraded, manual workaround exists) - **Low**: Pages - PR Staging - Cleanup, Pages - PR Staging - Sweep Stale, PR - Rebase, PR - Artifacts Comment, Merge Message, Track - Artifact Sizes, Track - Benchmarks, Sync - Agentic Data (broken = cosmetic/housekeeping) GitHub Actions failures don't block releases directly (AzDO owns that), but they indicate broken automation that accumulates tech debt if ignored. Flag all failures as actionable — the goal is "what automation is failing" not just "can we release". ### 2.9 Top Recommendations Provide **at most 5** prioritized actions, ordered by: 1. Blocks a release → highest priority 2. Breaks main → high 3. High-severity GitHub Actions failure → high 4. Chronic failure → medium 5. Medium-severity GitHub Actions failure → medium 6. Flake → low 7. Warning / low-severity GitHub Actions → informational Each recommendation should include: - What to do (imperative sentence) - Why (which branch/pipeline/workflow is affected) - Link to the relevant build or run --- ## Step 3: Assemble Report JSON After completing the analysis, assemble a **single JSON file** that combines the raw data with your analysis. This JSON is the source of truth for rendering. Write the JSON to: `output/ai/ci-status-YYYY-MM-DD.json` The schema is documented in `references/report-schema.md`. Top-level keys: ```json { "meta": { "date", "timestamp", "schemaVersion": "1.0", "window", "branches" }, "verdict": { "status", "emoji", "summary" }, "azdoHealth": { "branches": [...], "regressions": [...] }, "chainAnalysis": [ { "branch", "verdict", "summary", "rootPipeline", "cascadedPipelines" } ], "rootCauses": [ { "id", "title", "category", "severity", "footprint", "sampleError", ... } ], "githubActions": { "workflows": [...], "summary": { "total", "healthy", "failing", ... } }, "flakes": [ { "branch", "pipeline", "pattern", "confidence", "description" } ], "releaseRisk": [ { "branch", "shippable", "daysSinceGreen", "blockers", "recommendation" } ], "recommendations": [ { "priority", "severity", "action", "reason", "target", "buildUrl" } ] } ``` **Rules for assembling the JSON:** 1. The `azdoHealth` section contains the raw pipeline data from `ci-status-data.json`, restructured to match the schema 2. The `githubActions` section combines raw workflow data with your severity/status classifications