release-fleet-ops

name

release-fleet-ops

description

Drive and diagnose release→consumer fleet changes from inside arthur-debert/release: shipping a fix to the fleet (verify → canary run → cut — the cut refuses without green canary statuses on HEAD, auto-advances the major; consumers pull it), migrating/seeding a consumer onto the pull model, advancing the floating major, and — above all — diagnosing why a consumer's CI/gate is red and routing it to the right repo. Use when doing release-side work that affects consumers, or whenever you face 'is this a release bug or a consumer bug?'. Triggered by: release-core admin repos verify, release-core admin release advance-major, migrating a consumer, a consumer CI failure after a sync, or a fleet-wide lint/gate failure.

release-fleet-ops

The operating rules for release-side work that touches the fleet. release/ produces the files consumers run and the config that lints them, so most fleet failures are release bugs wearing a consumer's error message. This skill encodes the loop that routes them correctly — written because the obvious reflexes (fix the red consumer PR, spin an agent per repo) are wrong and waste hours.

The one rule: upstream-first

A consumer failure is upstream (a release/ bug) until proven consumer-specific. Release produces the distributed files (templates/**) and the gate config (commons). If a synced file fails a consumer's gate, the file or the gate is release's — fix it once in release/, not N times in consumers. Only failures in consumer-authored content are consumer-specific.

The default reflex is the opposite (the red thing is the consumer's PR, so fix it there). Resist it. Every fleet failure in the #348 saga — the gh-task-status shim, setup-dev-env.sh SC2015, the gate divergence — was upstream, and every consumer-side fix was wasted motion.

The loop

1. Reproduce once, in one throwaway clone. Not an agent per repo. The fleet is already cloned by release-core admin repos verify under /tmp/release-fleet-verify-$USER/. Reset one to clean main, install the managed files from your candidate ref (release-core init --no-commit with RELEASE_HOME/RELEASE_REF — the retired release-sync verb is gone), run the gate. One repo tells you what 15 would.
2. Consult the oracle — is it upstream? Two cheap, deterministic signals:
   • Is the failing file release-managed? A path that is a symlink into .release/ (or lives in templates/**) is release's. Consumer-authored files (their app-bin/, their src/, their Dockerfile) are theirs.
   • Does release's dogfood catch it? Release CI lints its own distributed output (the templates-bin-shellcheck job + the gate-unified.bats dogfood). If the failure reproduces against release's own tree, it is upstream by definition. Red dogfood = release bug. Green dogfood + red consumer = consumer-specific.
3. Route.
   • Upstream: fix in release/, open a PR, merge to main, run the pre-flight (release-core admin repos verify), run the canary round (release-core admin canary run --ref main — it stamps a canary/<family> commit status on the candidate sha), then cut a release — release-core cut REFUSES unless every registered canary/<family> status is green on the exact main HEAD being cut (no skip flag, release#606). release.yml publishes the wheel AND auto-advances the floating major (advance-major: true), so the cut IS the ship; there is no separate advance step to verify in front of. The fix reaches consumers by PULL — each self-updates at its next SessionStart. There is no push step (orc propagate was removed); the wheel is the carrier.
   • Consumer: fix in the consumer repo — but first rule out a shadow (below). Genuinely-consumer-authored content debt is the only thing that belongs in a consumer PR.
4. Pre-flight + canary before the cut; verify faithfully after. The sweep and the canary round must run BEFORE cutting (the cut auto-advances — there is no between window), and the cut enforces the canary half mechanically: the gate reads the canary/<family> statuses on the exact main HEAD, so a push after the round invalidates it by construction. After an upstream fix ships, a consumer's own next-session pull + its PR CI is the real gate — run what production runs (see "faithful pre-flight"), and verify with a fresh event, never gh run rerun (see "the rerun trap"). To migrate a consumer now rather than wait for its next session, run the resolver once in that repo (below).

The shadow trap (check this before any consumer fix)

A consumer can override the managed gate via lefthook-local.yml (lefthook merges it on top of the synced lefthook.yml). These files are usually stale workarounds someone hand-rolled for past gate brittleness, and they silently shadow the fixed upstream gate — so an upstream fix appears not to land.

When a consumer stays red after an upstream fix that should have worked:

lefthook dump | grep -A6 '<the failing step>:'   # the EFFECTIVE config
ls lefthook-local.yml .lefthook/ 2>/dev/null     # the shadow

If lefthook dump shows a command that is not the synced one, a local override is shadowing it. The fix is to delete the obsolete lefthook-local.yml (it exists only because the gate used to be brittle), not to re-patch upstream.

The tools (use these, don't reinvent)

• release-core admin repos verify --ref <ref> — hermetic pre-flight: clones the fleet, syncs each from <ref>, runs the gate. Use it BEFORE release-core cut — the cut auto-advances the floating major, so there is no later checkpoint. Its clones double as your reproduction sandbox. Expect npm/frontend kinds to FAIL on typecheck/eslint/prettier in the bare clones (missing-deps artifacts, not regressions — classify by failing step; #594 tracks making verify classify these itself).
• release-core admin canary run --ref main — the deep pre-ship round: a synthetic consumer lives its full life (boot from source, install the managed files, gate, e2e, a real prerelease cut) against the candidate sha and a canary/<family> commit status lands on release@<sha>. NOT optional before a cut: release-core cut refuses unless every registered family's status is green on the exact main HEAD it dispatches (release#606, no skip flag — the refusal names this command as the next action).
• Migrate/seed a consumer (replaces orc propagate, which was removed): in the target repo on a fresh branch, run the resolver once — bash bin/install-release-core (use release's own bin/install-release-core if the consumer's is pre-fix and can't self-bootstrap). It pulls the latest wheel and a bare init fully installs + auto-commits the managed files. Then push and open the managed-sync PR; its CI is the gate. One repo at a time; after the first seed the consumer self-updates natively. No fleet-wide push. (Release-dev note: if RELEASE_HOME is set in your env, prefix the run with env -u RELEASE_HOME so init builds from the published wheel bundle — what the consumer actually pulls — not your local release checkout.)
• orc probe --yes <clone> "<eval prompt>" — spin ONE fresh agent to evaluate a repo's state and report. Use for a perspective check, not as a per-repo fixer.
• release-core admin release advance-major — manual/recovery fast-forward of the floating major to main (ff-only). Normally NOT needed: release.yml auto-advances at cut (advance-major: true); reach for the verb only when that job failed or for an out-of-band advance.

Faithful pre-flight

A pre-flight that runs the gate differently from production lies. lefthook run pre-commit --all-files and a real git commit do not behave identically (file selection, glob exclude honouring). Verify with the same invocation the consumer's CI uses, and prefer the dogfood (release CI running the shared gate over its own output) as the source of truth — its green equals the consumer's green by construction.

The rerun trap: gh run rerun does NOT re-resolve @vN

A reusable-workflow ref (uses: arthur-debert/release/.github/workflows/x.yml@v2) is resolved once, when the run is created. gh run rerun re-executes that original snapshot — so after a cut advances @vN, rerunning a consumer's old red run still exercises the pre-advance release ref and proves nothing about the fix (caught live on padz, epic #583).

Fleet verification needs a fresh event on the consumer:

git commit --allow-empty -m "ci: re-resolve @vN" && git push   # fresh push run
# or: gh workflow run <wf> --repo <consumer>                   # fresh dispatch

Only a run created after the advance resolves the new vN tip.

Anti-patterns (all observed; all costly)

• Per-repo fix agents. Heavyweight (clone + install + full gate + CI wait), most get rejected at the permission prompt, and they fix symptoms. Reproduce once, fix the root.
• Consumer-first patching. Opening consumer PRs for what is a release/ bug just produces red CI in N repos. Classify before touching anything.
• Glob whack-a-mole in the gate. Per-file exclude patterns (**/x) behave inconsistently across lefthook modes and can't catch extensionless non-shell files. Selection belongs in a content-based runner (bin/check-shell), not in glob/exclude.
• Trusting a non-faithful pre-flight. verify-fleet --all-files passing did not mean consumers would pass — it ran a different mode. Match production.
• Status churn. Re-querying state you already have instead of reporting it.

Worked example: the gh-task-status shim

Symptom: 15 consumer re-sync PRs red on shellcheck SC1071 over bin/gh-task-status. Wrong path taken: fix consumers, spin agents. Right path: bin/gh-task-status was a release-managed symlink (a Python shim release shipped — since retired in #476 in favor of the pip console-script) → upstream. The gate was shellchecking a non-shell file. Root fix: bin/check-shell selects shell by content so the shim falls out; release dogfoods it. One PR, merged, re-synced. The two consumers that still stayed red had stale lefthook-local.yml shadows — obsolete workarounds, deleted, not re-patched.