github-pipeline-audit

# GitHub + Build Pipeline Audit Confirm that nothing is outstanding across GitHub and CI, and that everything that *should* be current *is* current. ## Outcome Produce a **clean-state report**: one row per surface, each either `CLEAN` or with a precise, actionable finding. The audit is only complete when every surface in the checklist below has been positively checked — "I didn't see anything" is not a result, `CLEAN` requires having run the query. **The governing principle: a green PR check is not the same as a healthy pipeline.** Several of the most damaging conditions this repo has hit are *invisible* from the PR and issue lists — the Dependabot updater silently failing so no security PRs are ever raised, a stale dependency-graph snapshot generating dozens of phantom alerts, a local checkout hundreds of commits behind. Sections 2, 5 and 6 exist specifically to catch the failures that do not announce themselves. ## Prerequisites ```bash gh auth status --hostname github.com # needs repo scope bk api "user" >/dev/null && echo "bk CLI ok" ``` - `gh` CLI authenticated (or `GITHUB_TOKEN` with `repo` scope). - The locally-authenticated **`bk` CLI** for Buildkite. Do **not** use the Buildkite API tokens in AWS Secrets Manager for logs — they deliberately lack the `read_build_logs` scope and return `"doesn't have the read_build_logs scope"`. They are fine for build *state*, triggering and retrying. - The `gh` token typically **lacks `read:project`**, so `projectsV2` GraphQL queries fail. That is expected — report Projects as "not checkable with current token scopes" rather than as clean. Set once: ```bash R=mock-server/mockserver-monorepo ``` --- ## 0. Check the checkout is current — do this FIRST Every subsequent version claim depends on it. A stale checkout produces confident, wrong answers: it will report already-fixed vulnerabilities as open, and validating a merged fix against it yields a **false negative**. ```bash git fetch origin git rev-list --count master..origin/master # MUST be 0 git status -sb | head -3 ``` If non-zero, either pull or read every file via `git show origin/master:<path>` for the rest of the audit. State in the report which ref each version claim came from. > Seen in practice: a checkout 186 commits behind, where a merged contributor fix was > absent locally and would have been reported as "not applied". --- ## 1. Issues ```bash gh issue list --state open --limit 100 gh issue list --state open --json number,title,labels,createdAt,comments,assignees \ --jq '.[] | {n:.number, t:.title, labels:[.labels[].name], age_days:(((now - (.createdAt|fromdate))/86400)|floor), comments:(.comments|length)}' ``` For each open issue establish: is it triaged (labelled), is it valid, and is anything actually in flight for it? Check for a linked PR — an issue with no PR and no branch is *unstarted*, not *blocked*, and should be reported that way: ```bash gh api repos/$R/issues/<N>/timeline --paginate --jq '.[] | select(.event=="cross-referenced" or .event=="connected")' gh pr list --search "<N>" --state all ``` Use the `issue-review` skill for a full validity assessment of any individual issue. **Do not trust an issue's own diagnosis.** Validate the reporter's stated root cause against the code and the cited standard — reporters are often right that something is broken and wrong about why, and the real constraint may be stricter or looser than the one they name. ## 2. Pull requests ```bash gh pr list --state open --limit 100 --json number,title,isDraft,author,createdAt,mergeable,reviewDecision,statusCheckRollup \ --jq '.[] | {n:.number, t:.title, draft:.isDraft, a:.author.login, mergeable:.mergeable, failing:([.statusCheckRollup[]? | select(.conclusion!="SUCCESS" and .conclusion!=null) | (.name//.context)+"="+(.conclusion//.state)])}' ``` For every PR that is not green, determine the root cause before recommending anything. **Diagnosis order — check these in sequence, because each rules out the next:** 1. **Is master itself red at the merge-base?** Dependabot PRs in this repo have repeatedly failed purely because they *inherited* a broken master. Always check master's own recent builds before blaming the PR. 2. **Is it infrastructure?** Buildkite `exit -1`, `"agent lost"`, or `"cancellation signal"` means **EC2 Spot reclamation**, not a test failure. Retry, don't debug. 3. **Are the many failures actually one failure?** A single Maven error cascades: the `Analyze (java)` CodeQL check runs Maven and fails identically, and every downstream client pipeline (Node/Python/Ruby/Go/.NET/Rust/PHP) fails `exit 1` because the Java artifact it waits on never built. **Count root causes, not red checks.** 4. **Only then**: the change itself. **Fork PRs get no Buildkite build** — the pipeline does not run for forks. To test one, push the SHA to an in-repo branch. ## 3. Security alerts — all three types These are three separate systems with separate APIs. Check all three; a repo can be clean in one and not the others. ```bash # Dependabot (dependency vulnerabilities) gh api repos/$R/dependabot/alerts --paginate \ --jq '.[] | select(.state=="open") | "\(.number)\t\(.security_advisory.severity)\t\(.dependency.package.name)\tscope=\(.dependency.scope)\trange=[\(.security_vulnerability.vulnerable_version_range)]\tpatched=\(.security_vulnerability.first_patched_version.identifier // "none")\t\(.dependency.manifest_path)"' | sort -n # Code scanning (CodeQL) gh api repos/$R/code-scanning/alerts --paginate \ --jq '.[] | select(.state=="open") | "\(.number)\t\(.rule.security_severity_level)\t\(.rule.id)\t\(.most_recent_instance.location.path)"' # Secret scanning gh api repos/$R/secret-scanning/alerts \ --jq '.[] | select(.state=="open") | "\(.number)\t\(.secret_type_display_name)\t\(.html_url)"' ``` ### Triaging a Dependabot alert — never act on severity alone For each alert, resolve these four facts before deciding anything: | Fact | How | Why it decides the outcome | |---|---|---| | **Resolved version** | `mvn dependency:tree`, or the lockfile on `origin/master` | The alert names a package, not your version | | **Vulnerable range** | `.security_vulnerability.vulnerable_version_range` | Your version may already be outside it | | **First patched version** | `.security_vulnerability.first_patched_version.identifier` | `null` = the branch is dead and unpatchable | | **Scope** | `.dependency.scope` (`runtime` / `development`) | Dev/test-only is materially lower risk | **Two stale-alert patterns account for most noise in this repo:** - **Dead-branch advisories.** A range like `<= 5.3.39` with `first_patched_version: null` against a tree running 7.x means the advisory can never apply — the dependency graph is a snapshot from before a major-version migration. Twenty Spring alerts were exactly this. Dismiss as `inaccurate`. - **Already at the patched version.** When the resolved version *equals* `first_patched_version`, the alert is simply stale. Dismiss as `inaccurate`. Prove scope claims — point at the pom `<scope>` or the lockfile `"dev": true` entry, and check whether the artifact reaches the shipped uber-jar or Docker image. Where you cannot prove non-applicability, treat it as REAL. ### Dismissing ```bash gh api repos/$R/dependabot/alerts/<N> -X PATCH \ -f state=dismissed -f dismissed_reason=inaccurate \ -f dismissed_comment='<=280 chars: version present, why the range does not apply, evidence>' ``` - **`dismissed_comment` is capped at 280 characters** — longer bodies fail with HTTP 422. Write the justification tight: version, why it does not apply, where you verified it. - Valid reasons: `fix_started`, `inaccurate`, `no_bandwidth`, `not_used`, `tolerable_risk`. - Dismissal is reversible; alerts can be reopened. - Never dismiss on a hunch — record the evidence in the comment so the next auditor does not have to redo the analysis. ## 4. Dependabot's *own* health — the silent failure **This is the check nobody thinks to run, and its absence is the most expensive gap.** When the Dependabot updater fails, no PRs are raised at all — the PR list looks calm precisely *because* the system is broken. ```bash gh run list --limit 40 --json databaseId,name,conclusion,createdAt \ --jq '.[] | select(.conclusion=="failure") | "\(.databaseId)\t\(.createdAt)\t\(.name)"' gh run view <id> --log-failed 2>&1 | grep -iE "error|Errors|message" | tail -40 ``` Ecosystem jobs appear as `<ecosystem> in <dirs> - Update #NNN`. Every one that is consistently failing is an ecosystem receiving **no updates and no security PRs**. Known causes seen here: | Symptom | Cause | Fix | |---|---|---| | `No pom.xml!` from `group_update_all_versions.rb` | Dependabot's `maven-wrapper-updater` experiment cannot parse a legacy Takari-format `.mvn/wrapper/maven-wrapper.properties` | Modernise the wrapper properties (`wrapperVersion`, `distributionType`, `distributionUrl`) or remove the wrapper | | `Could not determine Maven Wrapper version` | `maven-wrapper.properties` has only `distributionUrl` | Add `wrapperVersion` + `distributionType` | | `npm error code EOVERRIDE` | Dependabot injects an `overrides` entry that collides with a direct/transitive constraint | Pin the dep in that directory's `package.json`, or `ignore` the major | | A whole multi-directory group fails | **One bad directory aborts the entire grouped job**, blocking the healthy directories too | Split the failing directory into its own `package-ecosystem` block | | A directory silently never updates | Its manifest is **gitignored** — Dependabot reads the git tree, not the disk | Commit the manifest or drop the directory from `directories:` | **A clean onset date with no corresponding repo change is the signature of a Dependabot updater-image rollout, not a repo regression.** Correlate the first failure date against `git log` before hunting for a commit that does not exist. ## 5. GitHub Actions ```bash gh workflow list --all gh run list --branch master --limit 30 --json databaseId,name,conclusion,status,createdAt \ --jq '.[] | "\(.databaseId)\t\(.conclusion//.status)\t\(.name)"' ``` Distinguish **project builds** from **Dependabot updater jobs** in this list — they look alike and mean entirely different things. A `CodeQL` failure on a PR is usually the Maven build failing inside CodeQL's autobuild, not a CodeQL problem; read the Maven error. ## 6. Buildkite ```bash bk api "pipelines?per_page=50" | python3 -c "import sys,json;[print(p['slug']) for p in json.load(sys.stdin)]" # latest master build state per pipeline bk api "pipelines/<slug>/builds?branch=master&per_page=1" \ | python3 -c "import sys,json;d=json.load(sys.stdin);print(d[0]['number'],d[0]['state'],d[0]['web_url']) if d else print('none')" ``` Reading a job log (note: `bk api` prepends the org path — pass a **relative** endpoint): ```bash bk api "pipelines/<slug>/builds/<N>/jobs/<JOB_ID>/log" \ | python3 -c "import sys,json; print(json.load(sys.stdin).get('content',''))" \ | sed 's/\x1b\[[0-9;]*m//g; s/_bk;t=[0-9]*//g; s/\r//g' ``` The umbrella `mockserver` pipeline triggers the per-language pipelines; a red umbrella usually means one upstream Java failure, so start at `mockserver-java`. Driving the Buildkite UI through browser automation does **not** work for logs — that browser profile is logged out. Use the `pipeline-investigation` skill for a deep dive on any failure. ## 7. Branches and worktrees ```bash gh api repos/$R/branches --paginate --jq '.[].name' | grep -v '^master$' gh pr list --state all --json number,headRefName,state --jq '.[] | .headRefName+" "+.state' git worktree list ``` Report as outstanding: - **Remote branches with no open PR** — either merged-and-unpruned, or abandoned work. Confirm with `git cherry master origin/<branch>` whether the commits actually landed before suggesting deletion. - **Local worktrees** from finished sessions. `git worktree list` is authoritative. > Safety: `git status` **cannot** see gitignored `docs/plans/*.local.md`, so a worktree > can look clean while holding unsaved planning work. Check for those explicitly before > recommending removal, and never remove a worktree with a live session in it > (`lsof -d cwd`). ## 8. Discussions, wiki, projects ```bash gh api graphql -f query='{repository(owner:"mock-server",name:"mockserver-monorepo"){ discussions(first:30,states:OPEN){totalCount nodes{number title isAnswered category{name} createdAt}}}}' ``` Flag unanswered Q&A discussions — they are user-facing and easy to miss because they do not appear in the issue list. ## 9. Releases and published-artifact parity Confirm the last release actually reached **every** channel. Partial publication is a recurring failure mode: release-only code paths cannot be exercised by CI, so faults surface only during a release. ```bash gh release list --limit 5 ``` | Channel | Check | |---|---| | Maven Central | `curl -s https://repo1.maven.org/maven2/org/mock-server/mockserver-netty/maven-metadata.xml \| grep -E '<latest>\|<release>'` | | npm | `npm view mockserver-client version` / `npm view mockserver version` | | PyPI | `curl -s https://pypi.org/pypi/mockserver-client/json \| python3 -c "import sys,json;print(json.load(sys.stdin)['info']['version'])"` | | RubyGems | `curl -s https://rubygems.org/api/v1/gems/mockserver-client.json \| python3 -c "import sys,json;print(json.load(sys.stdin)['version'])"` | | Docker Hub | `curl -s "https://hub.docker.com/v2/repositories/mockserver/mockserver/tags?page_size=10"` | | Helm / Artifact Hub | chart `version` and `appVersion` MUST equal the app version |