release

# Release Process Releasing a new version is a multi-step process across several repositories. **IMPORTANT: Drive the ENTIRE process end-to-end in a single conversation.** Use the Bitrise MCP server to monitor build statuses (poll every 30s), check for triggered workflows in downstream repos, and move to the next step as soon as the previous one completes. Do not stop and wait for the user between steps. ## ✅ Definition of done — a release ships through MULTIPLE channels **A release is NOT done until every channel below is MERGED and VERIFIED — not merely opened.** Reporting "released" / "the steps are out" off a PR that *exists* but hasn't merged is the exact failure this gate exists to prevent. A just-shipped feature that isn't appearing where you expect is most often a release that didn't actually finish a channel — suspect that before rollout-timing or usage explanations. When you report status, report it **per channel**, and state what you actually confirmed (merged/verified) vs. what is still open. "The bot will handle it" is not done — go look. The channels: 1. **CLI GitHub release** — promoted out of prerelease, all expected assets present (Step 6). 2. **`verify-release`** — green (Step 6). 3. **`bump-prebooting` PR** (preboot VM image) — **merged** (Step 6b). This is how provision-injected features, e.g. the gradle-mirrors init script, reach the *default* fleet. The bypass-merge can stall and need a manual approval. 4. **Step auto-update PRs in all FIVE consumer repos** — **merged** (Step 7). 5. **Step GitHub releases** — cut for the scoped steps (Step 8). 6. **Steplib PRs** — merged (Step 9). 7. **Steplib spec published** — the new version is actually in the published spec (Step 10). A merged steplib PR does NOT mean the step shipped. Two distinct delivery paths exist and a complete release must finish BOTH: the **default fleet** gets CLI-driven features via **provisioning/preboot** (channel 3); customers who **pin a CLI version** get them via the **steps** (channels 4–7). Confirming one says nothing about the other. ## ⚠ Critical path — read before doing anything The `install/installer.sh` script and the binaries attached to every CLI GitHub release are **on the critical path of every Bitrise build — not just builds that opt into the build cache**. The Bitrise default workflow runs the gradle-mirrors activation step (and other CLI-driven steps) unconditionally, and each of those pipes `installer.sh` to `sh` and fetches the platform tarball + checksum from the latest non-prerelease GitHub release. If any of these break — installer script, binaries, checksum file, the wrong release marked as latest — the CLI install fails, the mirror activation soft-fails, and Maven Central requests bypass the Bitrise proxy on the entire fleet. That failure mode caused the [2026-04-28 Maven Central rate-limit incident](https://bitrise.atlassian.net/wiki/spaces/INCIDENT/pages/4980998155/2026-04-28+-+Postmortem+for+incident-2026-04-28-mavencentral-too-many-requests-5238). Concrete rules: - **Always create the CLI GitHub release as `--prerelease`** (see step 6). The installer ignores prereleases when resolving `latest`, so an empty / half-uploaded release cannot poison builds. - **Never let an empty release be marked `latest`.** If goreleaser fails midway, leave the release as prerelease until you have manually verified the assets list is complete. - **Treat any failure of the CLI `release` workflow as a drop-everything-and-fix incident.** Don't move on to step releases until the CLI release workflow is green AND the v2.6.x release has all 8 expected assets (6 platform tarballs, checksums.txt, both verification XMLs). - **Smoke-test installer.sh edits on a real Bitrise build before merging.** The release flow does not regenerate this file. PR CI does exercise it (see `pr-release-check-{linux,mac}` workflows) but a real Bitrise build is still the canonical smoke test. - **The release workflow mirrors `install/installer.sh` to GAR** twice: once as `installer.sh:<tag>:installer.sh` (immutable, audit trail) and once as `installer.sh:latest-pointer:installer.sh` + `installer.sh:latest-pointer:VERSION` (mutable pointer + bare semver, refreshed via delete-then-upload each release). GAR rejects the literal `latest` as a reserved version_id, so the mutable view uses `latest-pointer`. The `latest-pointer` view is the documented carve-out from the `#327` immutability rule — safe because it's only consulted when the primary GitHub path is already failing. After every release, the **`verify-release` workflow** runs (chained after `release` via the `release-and-verify` pipeline) and executes `scripts/verify_release.sh` to assert the GH happy path and the GAR-only fallback both work end-to-end. Verification failures post to Slack and can be retried independently of `release` without re-cutting the tag. ### Brew tap is best-effort, NOT critical path `bitrise-io/homebrew-bitrise-build-cache` is a nice-to-have publication target, not part of the install flow used by Bitrise builds. The "Publish Homebrew formula" step in `bitrise.yml`'s `release` workflow runs goreleaser with only the brew publisher and is marked `is_skippable: true`, so a tap-permission failure (e.g. 404 from `PUT /Formula/bitrise-build-cache.rb`) won't break the release. If the brew step fails: - Confirm the release workflow's overall status is still green. - Verify the GitHub release has all 8 expected assets, the GAR mirror uploads (binaries + checksums + `installer.sh:<tag>` + `installer.sh:latest-pointer:{installer.sh,VERSION}`) succeeded, and the chained `verify-release` workflow (running `scripts/verify_release.sh`) passed — all critical. A `verify-release` failure does NOT mean re-cut the tag — fix the underlying issue and re-run the `verify-release` workflow alone via the Bitrise UI. - Fix the tap separately. The most common cause is the bot user behind `GIT_BOT_USER_ACCESS_TOKEN` not being a collaborator with push access on `bitrise-io/homebrew-bitrise-build-cache`. Add them in the tap repo's settings. - Do NOT block other step releases on this — re-running the brew publish is a separate concern. ## Two Entry Points A release can be triggered by: 1. **Gradle plugin update:** A change merged in `bitrise-io/gradle-plugins` triggers the publish pipeline, which auto-triggers the CLI's `update_plugins` workflow. **Start from Step 1.** 2. **CLI-only changes:** Direct code changes to the CLI itself (e.g., Xcode fixes, new features) merged via a normal PR. The gradle-plugins steps don't apply. **Start from Step 5** (the CLI PR is already merged). ## Key Resources | App | Bitrise App ID | GitHub Repo | |-----|---------------|-------------| | gradle-plugins | `cdb16849-294e-48c4-8623-18ade050bd0e` | `bitrise-io/gradle-plugins` | | bitrise-build-cache-cli | `1a2ddc0a-bab0-4db1-9b78-4c13aae180ba` | `bitrise-io/bitrise-build-cache-cli` | | Gradle step (unified CI) | `48fa8fbee698622c` | `bitrise-steplib/bitrise-step-activate-gradle-remote-cache` | | Xcode step (unified CI) | `48fa8fbee698622c` | `bitrise-steplib/bitrise-step-activate-build-cache-for-xcode` | | Gradle features step (unified CI) | `48fa8fbee698622c` | `bitrise-steplib/bitrise-step-activate-gradle-features` | | React Native features step (unified CI) | `48fa8fbee698622c` | `bitrise-steplib/bitrise-step-activate-react-native-features` | | Gradle mirrors step (unified CI) | `48fa8fbee698622c` | `bitrise-steplib/bitrise-step-activate-gradle-mirrors` | | Steplib | — | `bitrise-io/bitrise-steplib` | ## Steps ### 1. Merge PR in gradle-plugins (skip if CLI-only change) Merging a PR to `main` in `bitrise-io/gradle-plugins` triggers the `publish-all` pipeline on the gradle-plugins Bitrise app. Only release modules that have version bumps in their `gradle.properties`. ### 2. Publish workflows auto-skip unchanged modules (skip if CLI-only change) Each publish workflow automatically checks if its module's `VERSION_NAME` was bumped in the current commit. If not, it skips publishing and exits successfully. No need to manually abort workflows. ### 3. Monitor publish builds (skip if CLI-only change) Poll the publish build(s) every ~30 seconds until they complete. Report status to the user. ### 4. Kick off CLI update workflow (skip if CLI-only change) The publish pipeline should automatically trigger an `update_plugins` workflow in the CLI app `1a2ddc0a-bab0-4db1-9b78-4c13aae180ba`. Check for running builds in that app. If the workflow wasn't triggered, manually trigger `update_plugins`. ### 5. Monitor CLI update and merge PR (skip if CLI-only change — PR is already merged) The `update_plugins` workflow creates a PR in `bitrise-io/bitrise-build-cache-cli`. Monitor the CI pipeline. If there are flaky cache hit rate failures, rebuild them (see "Flaky E2E tests" below). Once all checks pass: ```bash gh pr review --approve --repo bitrise-io/bitrise-build-cache-cli <PR_NUMBER> gh pr merge --merge --auto --repo bitrise-io/bitrise-build-cache-cli <PR_NUMBER> ``` **NEVER use `--admin` to bypass checks — always wait for CI to go green before merging.** ### 6. Create CLI GitHub release Create a GitHub release in `bitrise-build-cache-cli`. - **MUST mark it as `--prerelease`.** The release workflow uploads the binaries asynchronously; until those land, the release is empty. Marking it as latest (or as a regular release) at create-time makes a binary-less release "current," which breaks any consumer that downloads the latest asset. A separate CI job promotes the release out of prerelease once the binaries are appended. - **Do NOT pass `--latest`.** Without `--latest`, GitHub will not auto-promote a prerelease; with `--latest` it would, defeating the prerelease gate. - Example: `gh release create vX.Y.Z --repo bitrise-io/bitrise-build-cache-cli --title vX.Y.Z --prerelease --notes "..."` - Follow the format of existing releases for release notes - **Version numbering — always ask the user** which semver bump to apply (patch, minor, or major). Use these guidelines as defaults: - **Patch** bump: dependency-only updates (e.g., plugin version bumps) or bug fixes - **Minor** bump: new features or non-breaking behavioral changes in the CLI - **Major** bump: breaking changes - If this is a gradle-plugin-only update (no CLI code changes), the CLI version should be a **patch** bump because only a dependency was updated - Check the latest existing release tag to determine the next version ### 6b. Verify the preboot bump (`bump-prebooting`) The `release-and-verify` pipeline chains a `bump-prebooting` workflow after `verify-release`. It opens an auto-merging PR in `bitrise-io/build-prebooting-deployments` bumping `BITRISE_BUILD_CACHE_CLI_VERSION` + the per-arch sha256 in the two startup-script extensions, and tries to bypass-merge it as `bitrise-infrabot`. This is the channel that delivers provision-injected features (e.g. the gradle-mirrors init script) to the **default fleet** — a release that skips it ships to nobody on the default path. - **Confirm the PR actually MERGED — don't assume the bot finished.** The bypass-merge can fail (`base branch policy prohibits the merge`) even when the CLI release is fully green. - `gh pr merge --admin` does **not** work there (admins can't bypass that repo's rule). Fix: approve the bump PR by hand — it's authored by the bot, so a maintainer approval satisfies the 1-approval rule — then `gh pr merge --squash --delete-branch`. Verify the diff is the version + the two sha256s from this release's `checksums.txt`. Do NOT re-cut the tag. ### 7. Wait for step auto-update PRs The CLI release triggers auto-update PRs in **five** consumer repos. Monitor CI, then approve and merge each. All five are bumped by `bundle::update-step` in the `release` workflow (push-based, after the binaries publish), use unified CI app `48fa8fbee698622c`, and the PR title "feat: Release new CLI": 1. **Gradle step:** `bitrise-steplib/bitrise-step-activate-gradle-remote-cache` — released for every CLI version. 2. **Xcode step:** `bitrise-steplib/bitrise-step-activate-build-cache-for-xcode` — released for every CLI version. 3. **React Native features step:** `bitrise-steplib/bitrise-step-activate-react-native-features` — released, but releases are **not 1:1 with CLI releases** (each step release usually catches up across several intervening CLI patch versions; release when CLI changes matter for RN, e.g. an Xcode or Gradle-side improvement that RN builds benefit from). 4. **Gradle features step:** `bitrise-steplib/bitrise-step-activate-gradle-features` — truly experimental, no GitHub release flow yet (only a single early steplib PR exists). Merge the auto-update PR but do not cut a GitHub release until that changes. 5. **Gradle mirrors step:** `bitrise-steplib/bitrise-step-activate-gradle-mirrors` — used by customers who **pin a specific CLI version** (the default fleet gets the mirror init script via provisioning/preboot instead, so this step is the delivery channel only for pinned-version builds). It's a Go-module consumer (bumped via `go get`/`go mod tidy`/`go mod vendor`, same as the gradle features step). Now push-bumped by the `release` workflow like the others — historically it lagged badly (stuck at v2.6.1 / release 0.2.1 while the CLI was at v2.8.x) because it relied only on Renovate polling; the `bundle::update-step` push removes that lag. It uses 0.x step versioning, so cut a `0.x` GitHub release when you ship it. ```bash # For each step repo: gh pr review --approve --repo <REPO> <PR_NUMBER> gh pr merge --squash --auto --repo <REPO> <PR_NUMBER> ``` Always wait for CI to pass. Use `--squash` (merge commits are not allowed on these repos). ### 8. Create step GitHub releases Create GitHub releases for whichever of the five step repos the user actually wants to release (default: **Gradle step** + **Xcode step**; **React Native features step** when the CLI change is RN-relevant; **Gradle mirrors step** when its CLI bump matters for pinned-version users or it has fallen behind). The **Gradle features step** does not have a GitHub release flow yet — skip it. The **Gradle mirrors step** has its own 0.x release flow — cut a 0.x release after its bump PR merges. - These **can** be marked as "latest" - Follow the format of existing releases for release notes — only include "## What's Changed" with bullet points (changelog is added automatically) - **Version numbering: the step version bump should match the CLI version bump.** Patch CLI bump → patch step bump. Minor CLI bump → minor step bump. - For step repos that don't release for every CLI version (currently the RN features step), the changelog should note the headline change for this release **and** mention any intervening CLI versions that are being picked up at the same time — readers should be able to tell what they're getting. - Check the latest existing release tag in each repo to determine the next version - The user may explicitly scope the release to a subset of step repos ("only release xcode and rn-features"). Honor that — do not release the others. Merging their auto-update PRs is still fine and expected (keeps the dependency current); skipping is only about the GitHub release / steplib PR. ### 9. Merge steplib PRs After the step releases, PRs appear in `bitrise-io/bitrise-steplib` for each released step. They may need a rebase. ⚠ **steplib PR titles use the step's steplib ID, NOT its GitHub repo name — and TWO steps' names differ.** The steplib PR title is `<steplib-id>-<version>`. For most steps the id matches the repo, but two do NOT: the Gradle step's repo is `bitrise-step-activate-gradle-remote-cache` while its steplib id is **`activate-build-cache-for-gradle`**, and the React Native step's repo is `bitrise-step-activate-react-native-features` while its steplib id is **`activate-build-cache-for-react-native`**. So their steplib PRs are titled `activate-build-cache-for-gradle-<version>` and `activate-build-cache-for-react-native-<version>` — NOT `activate-gradle-remote-cache-…` / `activate-react-native-features-…`. If you search/poll for the wrong name the PR looks "missing" when it is in fact open (this cost ~40 min on the v3.2.2 release). To get a step's true steplib id, read the `Step id found:` line in its `release` build log (the `step-id-finder` step). The five repo → steplib-id mappings: | GitHub repo | steplib PR id | |---|---| | `bitrise-step-activate-gradle-remote-cache` | **`activate-build-cache-for-gradle`** | | `bitrise-step-activate-build-cache-for-xcode` | `activate-build-cache-for-xcode` | | `bitrise-step-activate-gradle-mirrors` | `activate-gradle-mirrors` | | `bitrise-step-activate-react-native-features` | **`activate-build-cache-for-react-native`** | | `bitrise-step-activate-gradle-features` | `activate-gradle-features` | **Merge each steplib PR as soon as it is individually MERGEABLE — do not batch-wait for all of them.** They appear and become mergeable at different times; gating on "all N present" stalls the whole release behind the slowest one (and behind any name-matching mistake above). ```bash gh pr review --approve --repo bitrise-io/bitrise-steplib <PR_NUMBER> gh pr merge --squash --auto --repo bitrise-io/bitrise-steplib <PR_NUMBER> ``` Always wait for CI to pass — never bypass branch protection. The steplib repo requires squash merges (merge commits are blocked). ### 10. Verify the steplib spec was published (⏳ SLOW — budget ~1h for a full batch) ⚠ **Tell the user up front that this step is a long wait.** Merging a steplib PR triggers a `deploy` build on **`tooling-steplib-controller [STEPLIB]`** (app `4320df4cdbe4bf24`, owned by Bitrise #Steps) which regenerates the spec every build resolves `@<major>` against. Those deploys are **serialized — one at a time, ~7–22 min each** (`pr_audit` builds are unaffected and start instantly). Merging four steplib PRs within a couple of minutes therefore queues four deploys back to back, and the last one waits out all the others. **The tail of a burst can be silently dropped.** A queued build that has not started within 60 minutes is killed by the idle reaper (`Idle Build: Build hasn't been started or updated since more than 60 minutes ago`), nothing retries it, and the spec keeps whatever the last *successful* deploy published — while PR merged / tag cut / steplib PR merged all still read green. The **last-merged step is the one at risk**, because its deploy sits at the back of the queue. v3.6.2 hit this exactly: seven deploys were queued from ~13:02, `activate-build-cache-for-react-native-0.14.2` merged at 13:18:51, and its deploy (#6094) was reaped at 14:20:45 — **71 seconds** before the deploy ahead of it finished at 14:21:56. The spec stayed at 0.14.1 and a customer on `@0` was still resolving CLI 3.6.1 seventeen hours later. Check the deploy queue (watch `queue_wait` against the 60-minute limit, and any `abort_reason`): ```bash curl -sS -H "Authorization: $BITRISE_PAT" \