multirepo-ci-cd

# Multi-Repo CI/CD Discipline Org-agnostic lessons for working across a portfolio of repos that share reusable workflows. The org-specific workflow catalogs live in the child rules (`*/.claude/rules/ci-cd-workflows.md`). ## Diagnosing a CI Failure — Fetch First Before investigating why a workflow failed, `git fetch` and compare local `HEAD` against `origin/main`. In a multi-repo portfolio, local checkouts go stale, and **the failing workflow file (or its config — `release-please-config.json`, the manifest, a new `.yml`) may have been added in commits you haven't pulled.** It won't exist in your local tree, so you'll search for a "missing" file that CI is actively running. One `git fetch` + `git log --oneline HEAD..origin/main` shows the gap; `git pull --ff-only` syncs it. Diagnose against what CI actually ran, not a stale snapshot. ## Rolling a Workflow Out to a Repo Class When promoting a self-contained workflow into a shared `reusable-*.yml` and adding a thin caller to every repo of a class, three things bite if missed: 1. **IaC feature flags are the source of truth for the repo set.** Don't hand-list repos — derive the class from the gitops/Terraform definitions (e.g. `release_please = true` entries in `repositories.tf` *are* the membership list). Extract them rather than guessing which repos qualify: ``` awk '/^ "[^"]+" = \{/ { match($0, /"[^"]+"/); name=substr($0, RSTART+1, RLENGTH-2) } /release_please[ ]*=[ ]*true/ { print name }' repositories.tf ``` 2. **Land the reusable workflow on the `.github` repo's `main` first.** Callers pin `uses: <owner>/.github/...@main`, which resolves at dispatch time — a caller merged or dispatched before the reusable workflow exists on `main` fails. Merge the `.github` PR before opening (or at least before merging) the callers. `workflow_dispatch`-only callers are harmless until triggered, so they can be opened in parallel, but the `.github` change is the prerequisite. 3. **Bulk-create callers via the GitHub API, not local clones.** The caller body is identical across repos, and some repos may not be checked out locally. Creating the branch + file + PR through the API is uniform and immune to local working-tree state (stale checkouts, wrong branch, uncommitted changes): ``` sha=$(gh api repos/<owner>/$r/git/ref/heads/main --jq .object.sha) gh api -X POST repos/<owner>/$r/git/refs -f ref=refs/heads/$BRANCH -f sha=$sha gh api -X PUT repos/<owner>/$r/contents/$PATH -f message=$MSG -f content=$B64 -f branch=$BRANCH gh pr create -R <owner>/$r --base main --head $BRANCH -a laurigates --title "$MSG" --body "$BODY" ``` Loop sequentially (not a parallel batch — siblings cancel on the first non-zero exit), and skip a repo if the file already exists on `main`. Title the caller PR `ci:` so release-please does not cut a version bump for a CI-only change. When the running gh user is the repo owner, omit `--add-reviewer` (self-review is rejected 422). ## Bulk-*Read* Through the API Too — a Stale Checkout Produces Wrong Findings The API-not-clones rule above is usually framed as a *write* discipline. It is equally a **read** discipline: a portfolio-wide `rg`/`grep` over local checkouts answers a question about *whatever each clone last fetched*, not about the repos. In a side-by-side layout where most clones are touched rarely, that answer is confidently wrong — and wrong in the direction of "this pattern isn't used anywhere," because a checkout predating an adoption simply lacks it. > Evidence (2026-07, ubuntu-slim runner sweep): a local `grep -rl` concluded > **0 repos** called `reusable-release-please.yml` and all 29 inlined > `googleapis/release-please-action` directly. That statement was made to the > user as fact. `pal-mcp-server` had migrated to the reusable on the remote > *weeks earlier*; the local clone was stale. Nothing about the grep output > looked suspicious. - **Read each file from origin in the same pass that patches it**: `gh api repos/<owner>/<repo>/contents/<path> --jq .content | base64 -d`. One fetch per repo is cheap and makes the finding and the edit agree. - **Use the local checkouts only to *generate candidates*, never to state conclusions.** Enumerate broadly locally (it's fast), then confirm every claim against the remote before reporting or acting on it. - **Make the patch function assert its precondition and raise.** The stale `pal-mcp-server` clone was caught only because the sweep's patcher demanded a `runs-on:` inside the named job and raised *"no runs-on found inside release-please job"* rather than falling through to a no-op or a mangled file. A silent `if found: patch` would have skipped it invisibly — and the wrong finding would have stood. Same instinct as `git-hazards.md`: a green run is not proof the content is right. - Cross-check the repo set itself against the remote as well (`gh repo list <owner> --json name,isArchived,isFork`) — a local directory can be a fork, an archived repo, or have no remote at all. ## Landing a `@main` Reusable-Workflow Fix on an Existing PR — Re-Trigger, Don't Re-Run After you merge a fix to a `reusable-*.yml` on the `.github` repo's `main`, the open PRs whose callers pin `@main` do **not** pick it up by re-running the failed check. **A GitHub re-run replays the original run against the workflow resolution it was first dispatched with** — including the exact commit the `@main` reusable workflow resolved to at that time. So `gh run rerun` / "Re-run failed jobs" / the `rerun-failed-jobs` API all replay the *stale* reusable workflow and fail identically, even though `main` is now fixed. The tell: you read the re-run's log and it shows the *old* behaviour (the command you just replaced still running), not your fix. **Fix: force a fresh `pull_request` event**, which re-resolves `@main` at dispatch time and picks up the merged fix. Cheapest first: ``` gh pr update-branch <n> -R <owner>/<repo> # fires `synchronize`; no-op-safe if already current ``` If the branch is already current (update-branch reports "already up to date"), close-then-reopen fires `reopened` instead: ``` gh pr close <n> -R <owner>/<repo> && gh pr reopen <n> -R <owner>/<repo> ``` For a **release-please** PR, prefer `update-branch` — closing the release PR is safe (release-please acts on push-to-`main`/schedule, not PR-close events) but reopening is noisier than a branch update. Avoid pushing to a release-please branch by hand. There is a second replay trap with the same shape: a `pull_request` re-run also replays the original event payload, so its `sender` is whoever opened the PR — a **bot** for Renovate/release-please PRs. Actions with a human-actor guard (e.g. `anthropics/claude-code-action` without `allowed_bots`) therefore fail with "Workflow initiated by non-human actor" on re-run **regardless of who clicked re-run**. A fresh event doesn't change the sender (still the bot), so the real fix is `allowed_bots` on the action, not re-triggering — but the same "re-run replays the original payload" mechanic is why clicking re-run yourself doesn't help. Both traps are the same root cause: a re-run is a replay, not a new dispatch.