cutting-a-release

# Cutting a Release Single source of truth for cutting a release. Three concerns move together: the **branch/PR workflow**, the **version bump**, and the **two release documents** (`RELEASE_NOTES.md` + `CHANGELOG.md`). Do the steps in order, all on one `release/x.y.z` branch cut from `develop`. Detailed format templates live in reference files — load them when you reach §6: - Release-notes structure, section order, and spotlight template → [references/release-notes-format.md](references/release-notes-format.md) - Changelog structure and per-release entry skeleton → [references/changelog-format.md](references/changelog-format.md) --- ## 0. Shape of a release ``` develop (PR-only) main (PR-only, squash) │ 1. pull latest develop │ │ 2. cut release/x.y.z ────────────┐ │ bump VERSION + sync │ │ write CHANGELOG + NOTES │ │ commit + push │ │ 3. PR release/x.y.z ──▶ main (squash-merge after review) │ 4. backmerge ◀────────────────────┤ (branch off develop, merge main in, │ (PR to develop, MERGE COMMIT)│ PR into develop, MERGE COMMIT) ▼ ▼ ``` Two hard rules of this repo's branch model: 1. **Never commit directly to `develop` or `main`.** Both are PR-only — the release bump and the backmerge each land through a branch + PR. 2. **After the squash-merge into `main`, you MUST backmerge `main` → `develop`** (§7). Skipping it leaves the branches divergent and every future release conflicts on the release-artifact files. --- ## 1. Pre-merge prerequisite — the DynamoDB GSI limit **Ask before opening the release PR: does any table that ALREADY EXISTS in production gain more than one Global Secondary Index in this release?** If yes, the release **must be split into two deploys** — ship one index, wait for it to report `ACTIVE`, then ship the next. That split has to happen **before the merge**, not after a rollback. ### Why one is the limit DynamoDB's `UpdateTable` API permits **exactly one GSI creation or deletion per call**. CloudFormation issues one `UpdateTable` per changed table, so a template that adds two indexes to an existing table fails with: ``` Cannot perform more than one GSI creation or deletion in a single update ``` The limit applies to **`UpdateTable` only**. `CreateTable` accepts any number of indexes, so **a brand-new table with five GSIs is fine** — which is exactly why the question is "does an *existing* table gain more than one?" and not "are there new indexes?". The blast radius is the whole stack, not the table: when the update fails, CloudFormation rolls back **every other resource in the deploy** with it. Meanwhile `backend.yml` and `frontend-deploy.yml` are separate workflows that succeed on their own, so the release ends up running **new application code against old infrastructure**. ### Why an incremental environment cannot reveal this `dev` deploys on every merge into `develop`. Two indexes that arrive in two separate PRs therefore get **two separate platform deploys**, one index each — both succeed, and dev looks completely healthy. `main` only ever sees the *aggregate*. A release collapses a whole batch of merges into a single CloudFormation update, so the two indexes that dev applied one at a time arrive at prod **in the same `UpdateTable` call**. Only an environment that jumps a whole release at once is exposed, which means **no amount of dev soak time will surface it** and passing CI proves nothing here. Reviewing the release diff is the only place it can be caught. > **This is what took production down on 2026-08-01.** Release 1.12.0 (#814) > added `AgentDirectoryIndex` and `AgentReportsIndex` to the existing > `{prefix}-rag-assistants` table in one update. They had reached `develop` in > separate merges (`02d0f2e9`, `a128831d`) and dev had deployed each one > cleanly. The prod update failed, rolled back the rest of the deploy including a > brand-new audit-log table, and left the agent store returning 500 against > already-deployed new code. Recovery took two more patch releases. The 1.12.0 > release notes *did* flag GSI backfill timing — but not the per-update limit. ### How to check CI enforces this on every PR into `main` (`.github/workflows/gsi-update-limit.yml`). To check locally before you open the PR: ```bash node scripts/release/check-gsi-update-limit.mjs ``` It diffs `infrastructure/gsi-inventory.json` — a committed, synth-generated inventory of every DynamoDB table and its indexes — between `origin/main` and your branch, and fails when a table present in **both** needs more than one index operation. Tables that appear only on your branch are exempt (`CreateTable`), as are tables dropped entirely (`DeleteTable`). Note it counts **creations and deletions together**, because the API limit is one GSI *operation* per update: renaming an index — one add plus one drop — fails exactly like adding two does. The inventory is regenerated and verified by the infrastructure test suite, so a GSI added in code without the inventory updated fails `npx jest` first: ```bash cd infrastructure && UPDATE_GSI_INVENTORY=1 npx jest gsi-update-limit ``` > The check reports **SKIPPED** when `main` has no inventory to diff against — > true only until the first release carrying `gsi-inventory.json` lands. For that > one release, answer the question by reading the diff yourself. **The automation does not replace the question.** It compares committed inventories, so it only sees what `git` sees. Read the `gsi-inventory.json` diff in the release PR: one added line under an existing table is fine, two is a split. ### If the answer is yes — split the release Ship the indexes in separate releases, each merged and deployed on its own: 1. Release N carries **one** of the new indexes. Merge, deploy, and confirm the index reports `ACTIVE` before continuing — CloudFormation reporting `UPDATE_COMPLETE` is **not** the same as the index being usable: ```bash aws dynamodb describe-table --table-name <prefix>-rag-assistants \ --query 'Table.GlobalSecondaryIndexes[].{Name:IndexName,Status:IndexStatus}' ``` 2. Release N+1 adds the second index, once the first is `ACTIVE`. Prefer doing this **on the release branch before the merge**: drop the second index from the release branch, ship it, then restore it in the next release. ### Recovery, if it has already failed Each half needs its **own patch release** — there is no way to ship them as one PR and no way to skip the version bump. `.github/workflows/version-check.yml` fails any PR into `main` whose `VERSION` is unchanged, and it has **no label, flag, or skip path**. So a failed two-index deploy costs two patch releases (1.12.0 → 1.12.1 added the first index alone, 1.12.2 restored the second). > **Do NOT stack PR 2 on PR 1.** After PR 1 squash-merges into `main`, the > merge-base for a branch cut from PR 1 does not move. PR 1's removal and PR 2's > restoration then **cancel out in the three-dot diff GitHub shows and merges** — > the PR looks correct, CI passes, and the restore silently no-ops. The index > never comes back. > > Open PR 2 **only after PR 1 has merged**, branched fresh from `main`, as a > `git revert` of PR 1's squash commit: > > ```bash > git fetch origin main > git switch -c release/x.y.z origin/main > git revert --no-commit <pr-1-squash-sha> # restores the second index > ``` --- ## 1b. Pre-merge prerequisite — data backfills A backfill script is not code that runs itself. Someone has to run it, against each environment, in a window that matters. **If this release adds one, the release notes must name it** — that is the only place an operator looks. ### Why this is its own gate The failure is silent, and in the same shape as §1. Backfills populate a sparse index or a new attribute, and the read path does not error when the data is missing — a sparse index returns **fewer rows**, not an exception. An unrun backfill therefore looks like a short list, not an outage: the tool catalog missing half its tools, the artifact library missing the older entries. Nothing goes red, and no alarm fires. `develop` cannot surface it either. Whoever wrote the script ran it against dev by hand the day they wrote it, so dev is always fine. **Prod is the environment with nobody assigned**, and the release is the last moment the instruction can still reach someone. ### How to check CI enforces this on every PR into `main` (`.github/workflows/pending-backfills.yml`). Locally: ```bash node scripts/release/check-pending-backfills.mjs ``` It diffs `backend/scripts/backfill_*.py` between `origin/main` and your branch and fails when a script added in the range is not named in `RELEASE_NOTES.md`. It cannot verify the backfill was actually **run** — no CI job can know that. It guarantees the instruction reaches the person who can. ### What to write Name the script, the command, and the environments. Not "a backfill is required" — that is the note that sends someone digging through `git log`: > **Manual step — run before/with this deploy.** Populates `EntityTypeIndex` > for tool rows written before it existed. Idempotent; dry-run by default. > > ```bash > AWS_PROFILE=<env> backend/.venv/bin/python backend/scripts/backfill_tool_catalog_index.py \ > --table <prefix>-app-roles --region us-west-2 --apply > ``` > > Verify `skipped=0 failed=0`, then confirm with a live Query on the index. Two details that cost real time on the v1.21.0 prod run, both worth carrying into whatever backfill note you write: - **Give the backend venv's interpreter, not a bare `python`.** These scripts need `boto3`, which is not in the system Python — a bare `python` fails with `ModuleNotFoundError: No module named 'boto3'` before it does anything. All six `backfill_*.py` scripts are runnable as `backend/.venv/bin/python backend/scripts/<script>.py …` from the repo root (`uv run --project backend python …` works too). - **Never tell an operator to verify with `describe-table` `ItemCount`.** DynamoDB refreshes table and index item counts roughly every **six hours**, so a correct backfill still reports `0` immediately afterwards and reads as a failure. Name a live `query … --select COUNT` instead, and pair it with a scan for rows the backfill missed — a *partial* backfill is silent, and is exactly what a sparse-index read path's fallback does not cover. ### Ordering, when the release also switches a read onto the backfilled data Deploy → backfill → *then* the read. If the release carries both the backfill and the code that depends on it, the backfill runs **inside the release window**, not afterwards. A fresh deployment is exempt only if its seed path already writes the new keys — check, do not assume: `seed_bootstrap_data.py` hand-builds its items rather than going through the model. --- ## 2. Branch workflow ```bash git switch develop git pull --ff-only origin develop # release must contain all of develop git switch -c release/x.y.z # branch name == version being shipped (no v prefix) ``` The branch name **corresponds to the version being pushed** (`release/1.2.0` ships `1.2.0`) and carries **all commits currently on `develop`**. Then, on the release branch: create a task list, determine the bump (§4), bump `VERSION` + sync (§3), write both docs (§5–§6). Keep every edit on the release branch — `develop` stays untouched. Commit and push: ```bash git add VERSION backend/pyproject.toml backend/uv.lock \ frontend/ai.client/package.json frontend/ai.client/package-lock.json \ infrastructure/package.json infrastructure/package-lock.json \ CHANGELOG.md RELEASE_NOTES.md README.md git commit -m "Release/x.y.z" -m "<one-paragraph summary + bullet notes>" git push -u origin release/x.y.z ``` Open a PR `release/x.y.z` → `main`. After prerequisites (§1) + code review pass, it is **squash-merged** into `main` (the version-check gate passes because `VERSION` changed). Then do the backmerge (§7). --- ## 3. Version bump Single `VERSION` file at the repo root is the source of truth. Format: `MAJOR.MINOR.PATCH[-PRERELEASE]` (SemVer 2.0), no `v` prefix. 1. Edit `VERSION`. 2. `bash scripts/common/sync-version.sh` 3. `bash scripts/common/sync-version.sh --check` → must print `[PASS]`. 4. Commit `VERSION` **and every file the sync touched, including lockfiles**. The sync script propagates the version into `backend/pyproject.toml`, `frontend/ai.client/package.json`, `infrastructure/package.json`, the `README.md` badge + "Current release" line, and **regenerates** `backend/uv.lock` and both `package-lock.json`. Run it inside the dev container (needs `uv` + `npm` at pinned versions). - **PR gate:** PRs to `main` fail CI if `VERSION` is unchanged vs `main` or the manifests are out of sync. - **CI handles the rest:** Docker image tags, AWS resource tags, health-endpoint versions, frontend version display, and the `vX.Y.Z` git tag. Just bump + sync. --- ## 4. Decide the bump — SemVer + release sizing Pick the tier from what shipped, then size the write-up to match. | Bump | When | Notes depth | |---|---|---| | **Patch** `x.y.Z` | Bug fixes, security/dep bumps, CI/CD, docs, internal refactors — no new user-facing capability | **Brief.** 2–4 sentence Highlights + compact per-category bullets + one deployment note. No spotlights, no per-layer subsections. | | **Minor** `x.Y.0` | New features, endpoints, pages, capabilities (even if flag-gated/preview) | **Deep.** Highlights + one spotlight per major feature (backend/frontend/infra/tests) + per-category bullets. | | **Major** `X.0.0` | Breaking changes, architecture shifts, migrations | **Deepest.** Above + prominent migration/upgrade section and breaking-change callouts. | Rules of thumb: don't pad a patch; don't starve a feature; size to the largest single change; turning a dark feature **on by default** or completing a preview capability counts as user-facing → **minor**. --- ## 5. Identify what changed (the divergent-history trap) `develop` is squash-merged → `main`, so **`main` and `develop` have divergent histories**; a squash collapses a whole release into one commit on `main` whose SHA matches nothing on `develop`. > **Do NOT use `git log main..develop`.** Even after a backmerge makes `main` an > ancestor of `develop`, main's *squashed* history means develop's granular > commits are never reachable from main — so `main..develop` returns the entire > project history, not the release delta. Use the previous release's cut point (or tag) as the boundary: ```bash git tag --list 'v*' --sort=-creatordate | head git log <prev-release-cut-or-tag>..develop --no-merges --reverse --format='%h %s' git log <prev-release-cut-or-tag>..develop --merges --format='%s' # PR numbers ``` Exclude commits a prior **backmerge** dragged in (the previous `Release/x.y.z` squash commit, direct-to-main hotfixes) — they aren't new work. For every non-trivial commit **read the diff, not the message** (`git show --stat <sha>`). Bucket each change: | Category | Emoji | Use for | |---|---|---| | New | 🚀 | New features, endpoints, pages, capabilities | | Improved | ✨ | Enhancements to existing features | | Fixed | 🐛 | Bug fixes | | Changed | ⚠️ | Breaking changes, removals, deprecations, migrations | | Security | 🔒 | CVE patches, CodeQL fixes, auth hardening | | Performance | ⚡ | Latency/throughput/cost wins users notice | | Infrastructure | 🏗️ | CDK/IaC changes, new AWS resources, deploy-order changes | | Dependencies | 📦 | Package upgrades (grouped in a table) | | CI/CD | 🔧 | Workflow/pipeline/tooling changes | | Docs | 📚 | Documentation worth calling out | **Include** user/operator-facing changes, bug fixes people hit, security updates, breaking changes, dependency upgrades. **Changelog-only:** minor dep bumps with no behavior change, internal test additions. **Exclude from both:** pure internal refactors, typo/comment/formatter churn. --- ## 6. Write the two documents Same pass, same categorized list. Write `CHANGELOG.md` first (factual log), then `RELEASE_NOTES.md` (narrative). Cross-check every changelog line maps to something in the notes (except the changelog-only items above). - **`CHANGELOG.md`** — [Keep a Changelog], terse, one line per change, PR-linked `(#NNN)` when known, grouped by category emoji. New version at top; omit empty categories; never invent PR numbers. Full skeleton → [references/changelog-format.md](references/changelog-format.md). - **`RELEASE_NOTES.md`** — narrative, benefit-first with technical depth. New release at top; **never modify previous entries.** Lead each item with the user/operator outcome, then the mechanism (files, endpoints, classes). Section