cihealth

name

cihealth

description

Check the health of the CI/CD pipeline — devel branch status, per-commit checks, release artifacts, artifact pin staleness, image publishing, scheduled jobs, repinning cadence. Autonomously creates fix PRs for trivial issues (formatting, etc.) or proposes a diagnosis plan for deeper problems. Use when you suspect CI is broken, main is red, releases are lagging, repinning is stuck, or images are falling behind.

CI Health Check

Comprehensive CI/CD pipeline health audit for the agentydragon/ducktape repository. Gather data in parallel, then produce a single structured report. Autonomously fix trivial issues (e.g. prettier formatting failures); propose diagnosis plans for anything that needs deeper investigation.

Setup — GitHub Authentication

gh reads GITHUB_TOKEN. In a Claude Code web session, the read-only token is already available in the environment as DUCKTAPE_CI_READ_GITHUB_TOKEN (exported by devinfra/secrets/web_env.sh via the session start hook), so export GITHUB_TOKEN="$DUCKTAPE_CI_READ_GITHUB_TOKEN" before running gh commands. Outside that context, decrypt secrets/github-ci-read-pat.yaml with the SOPS age key and export it as GITHUB_TOKEN; see devinfra/secrets/web_env.sh for the exact pattern.

If auth is unavailable, continue with the public API (rate-limited) and note it in the report.

Phase 1 — Discover CI Organisation

Don't hardcode checks. Read what the repo actually defines:

# List all workflows
ls /home/user/ducktape/.github/workflows/

# Read devinfra/ci/artifacts.py to understand what gets released
cat /home/user/ducktape/devinfra/ci/artifacts.py 2>/dev/null | head -80

# Read npins/sources.json to see what's pinned
cat /home/user/ducktape/npins/sources.json

# Read devinfra/image_pins.json for Dockerfile-based image pinning
cat /home/user/ducktape/devinfra/image_pins.json 2>/dev/null

# Count commits on devel since last pin update for each artifact
git -C /home/user/ducktape log --oneline origin/devel | head -1  # current HEAD

From the discovery, build a mental model of:

• Which workflows run on push to devel (per-commit checks)
• Which workflows run on a schedule (repinning, attic push)
• Which workflows are triggered manually only
• What artifacts are released and pinned back into the repo

Phase 2 — Gather Data (Run in Parallel)

Run all data-gathering commands in parallel.

2a. Per-commit Workflow Status on devel

For each push-triggered workflow, fetch the last 5 runs on devel:

REPO=agentydragon/ducktape

# Per-commit workflows to check:
for wf in pre-commit ci bazel-ci ansible-lint nix-attic-push \
           push-images props-images container-images \
           openclaw-image tana-mcp-image; do
  echo "=== $wf ==="
  gh run list --repo $REPO --workflow "${wf}.yml" \
    --branch devel --limit 5 \
    --json headBranch,status,conclusion,displayTitle,createdAt,databaseId \
    --jq '.[] | [.conclusion, .createdAt[:16], .displayTitle[:60]] | @tsv' \
    2>/dev/null || echo "(no runs or workflow not found)"
done

For any workflow whose most recent run is not success:

• Read the failure log: gh run view <id> --repo $REPO --log-failed 2>&1 | tail -100
• Identify exactly what failed (hook name, step name, error message, diff)

2b. Release Artifact Staleness

For each entry in npins/sources.json, the URL encodes the pinned commit: ducktape-produced pins follow the pattern .../releases/download/<artifact>-<sha7>/<filename>. Extract that sha7, count how many commits origin/devel is ahead of it, and report the age of the pinned commit.

External pins (e.g. bb from buildbuddy-io) have semantic version tags instead — just report the version.

Also read devinfra/image_pins.json for Dockerfile-based image digests (rbe-worker, freecad-test). These are updated by container-images.yml only when the relevant Dockerfiles change, so staleness is expected unless those paths were recently touched.

2c. Scheduled Workflow Health

Check that scheduled jobs are running and succeeding:

REPO=agentydragon/ducktape

# sync-pins runs every 30 minutes — should have a recent success
echo "=== sync-pins (every 30 min) ==="
gh run list --repo $REPO --workflow sync-pins.yml \
  --limit 5 \
  --json status,conclusion,createdAt,databaseId \
  --jq '.[] | [.conclusion, .createdAt[:16]] | @tsv' 2>/dev/null

# nix-attic-push runs on push
echo "=== nix-attic-push (on push) ==="
gh run list --repo $REPO --workflow nix-attic-push.yml \
  --branch devel --limit 3 \
  --json status,conclusion,createdAt,displayTitle \
  --jq '.[] | [.conclusion, .createdAt[:16], .displayTitle[:50]] | @tsv' 2>/dev/null

# nix-flake-update is manual — just report last run
echo "=== nix-flake-update (manual) ==="
gh run list --repo $REPO --workflow nix-flake-update.yml \
  --limit 3 \
  --json status,conclusion,createdAt \
  --jq '.[] | [.conclusion, .createdAt[:16]] | @tsv' 2>/dev/null

sync-pins health criteria:

• Should have a success run within the last 45 minutes
• If last success is >60 min ago: warn; if >2 hours: flag as stuck
• If recent runs are failing: read failure log and diagnose

2d. Release Pipeline Status

Check that the release job ran and released all expected artifacts after the most recent devel push:

REPO=agentydragon/ducktape

echo "=== ci.yml (main pipeline) ==="
gh run list --repo $REPO --workflow ci.yml \
  --branch devel --limit 5 \
  --json status,conclusion,createdAt,displayTitle,databaseId \
  --jq '.[] | [.conclusion, .createdAt[:16], .displayTitle[:60]] | @tsv' 2>/dev/null

# Latest release artifacts on GitHub
echo "=== Latest GitHub releases ==="
gh release list --repo $REPO --limit 20 2>/dev/null \
  | head -30

Cross-reference: after the most recent successful ci.yml run, there should be GitHub releases for every artifact listed in devinfra/ci/artifacts.py. If any are absent or older than the CI run, the release job may have failed silently.

2e. Container Image Currency

REPO=agentydragon/ducktape

echo "=== container-images (rbe-worker, freecad-test) ==="
gh run list --repo $REPO --workflow container-images.yml \
  --branch devel --limit 5 \
  --json conclusion,createdAt,displayTitle \
  --jq '.[] | [.conclusion, .createdAt[:16], .displayTitle[:50]] | @tsv' 2>/dev/null

echo "=== push-images (14 OCI images) ==="
gh run list --repo $REPO --workflow push-images.yml \
  --branch devel --limit 3 \
  --json conclusion,createdAt,displayTitle \
  --jq '.[] | [.conclusion, .createdAt[:16]] | @tsv' 2>/dev/null

echo "=== openclaw-image ==="
gh run list --repo $REPO --workflow openclaw-image.yml \
  --branch devel --limit 3 \
  --json conclusion,createdAt \
  --jq '.[] | [.conclusion, .createdAt[:16]] | @tsv' 2>/dev/null

echo "=== tana-mcp-image ==="
gh run list --repo $REPO --workflow tana-mcp-image.yml \
  --branch devel --limit 3 \
  --json conclusion,createdAt \
  --jq '.[] | [.conclusion, .createdAt[:16]] | @tsv' 2>/dev/null

2f. Anything Else the Repo Defines

After running the above, glance at .github/workflows/ for any workflows not covered above. Spot-check their last run status.

Also check for signs of CI health issues beyond GitHub Actions:

# Are there uncommitted pin updates stuck in a PR?
gh pr list --repo agentydragon/ducktape --limit 10 \
  --json title,state,createdAt,headRefName \
  --jq '.[] | [.state, .createdAt[:10], .headRefName[:40], .title[:50]] | @tsv' \
  2>/dev/null | grep -iE 'pin|sync|chore.*sync' | head -10

Phase 3 — Diagnose Failures

For every failure found in Phase 2:

1. Read the failure log (already done for per-commit checks above). If not yet read, do it now: gh run view <id> --repo ... --log-failed 2>&1 | tail -150

2. Classify the failure:

   • Trivial / auto-fixable: prettier formatting, trailing whitespace, import ordering, minor linting — create a fix commit and open a PR
   • Test failure: a specific test broke — read the test output, identify the cause, decide if it's a one-line fix or needs deeper investigation
   • Infrastructure failure: runner can't reach a dependency, SOPS key missing, cache full, external API down — propose diagnosis steps
   • Pin/release stuck: artifact not being released or pinned — trace the release pipeline (ci.yml → release.yml → sync-pins.yml) to find the break

3. For trivial fixes: implement them directly, commit, push a branch, and open a PR targeting devel. Report the PR URL in the health report.

4. For non-trivial issues: write a diagnosis plan with specific commands the user can run to confirm the root cause, followed by proposed fixes.

Phase 4 — Report

Produce a single markdown report. Healthy items get one line. Issues get details.

# CI Health Report — <date> <time UTC>

## Summary

<one-liner: all green / N issues found>

## Per-Commit Checks (devel)

| Workflow         | Status | Last run         | Note                     |
| ---------------- | ------ | ---------------- | ------------------------ |
| pre-commit       | ✅     | 2026-04-13 06:19 |                          |
| bazel-ci         | ✅     | 2026-04-13 06:19 |                          |
| release          | ✅     | 2026-04-13 06:19 | all 6 artifacts released |
| push-images      | ✅     | 2026-04-13 06:15 |                          |
| ansible-lint     | ✅     | 2026-04-12 14:03 |                          |
| nix-attic-push   | ⚠️     | 2026-04-12 10:00 | 28h ago, timed out       |
| container-images | ✅     | 2026-04-11 08:00 | no Dockerfile changes    |
| <other>          | ...    | ...              | ...                      |

## Scheduled Jobs

| Job              | Schedule     | Last success        | Status |
| ---------------- | ------------ | ------------------- | ------ |
| sync-pins        | every 30 min | 2026-04-13 06:32    | ✅     |
| nix-flake-update | manual       | 2026-04-10 (manual) | ✅     |

## Artifact Pin Staleness

| Artifact      | Behind devel | Pinned commit | Age        |
| ------------- | ------------ | ------------- | ---------- |
| claude-hooks  | 0 commits    | 35b400e       | up-to-date |
| skills        | 2 commits    | 2de0bf8       | 1 hour ago |
| ducktape      | 0 commits    | 071dad5       | up-to-date |
| bbapi         | 0 commits    | 1710bc6       | up-to-date |
| bb (external) | n/a          | 5.0.339       | external   |

## Dockerfile Image Pins

| Image        | Pin age | Status |
| ------------ | ------- | ------ |
| rbe-worker   | current | ✅     |
| freecad-test | current | ✅     |

## Issues Found

### <severity>: <title>

: 
:  at 
: 
: 
: 

Important Patterns to Detect

• sync-pins not running: if last sync-pins.yml success is >1h ago, something is wrong with the scheduler or the workflow itself is failing
• Release blocked: if ci.yml succeeded but release.yml sub-job failed, artifacts won't update and pins will grow stale
• Artifact drift with healthy pipeline: pins can legitimately lag by several commits while a CI run is still in progress (commits often land while the release job is building). To assess whether drift is a problem, trace the pipeline: did ci.yml complete successfully for the commits since the pin? Did the release sub-job succeed for that specific artifact's test matrix entry? Did sync-pins.yml run after that release completed? If all three happened and the pin still hasn't moved, something is stuck upstream
• Prettier / formatting churn: if pre-commit fails on the same file repeatedly across different commits, the file likely has an ongoing formatting conflict (contributor not running pre-commit locally)
• Container image stuck: if container-images.yml last ran weeks ago and the Dockerfiles haven't changed, this is expected — only flag if the image should have updated based on path changes in recent commits
• nix-attic-push timeout: the job has a 120-minute timeout; timeouts are not uncommon for large Nix builds but should be retried