| name | release |
| description | Full release lifecycle for Kagenti — alpha, RC iteration loop, GA, and patch releases with multi-repo coordination |
flowchart TD
START(["/release"]) --> ASSESS["Phase 1: Assess"]
ASSESS --> PREPARE["Phase 2: Prepare & Tag"]
PREPARE --> VERIFY["Phase 3: Verify Artifacts"]
VERIFY --> STABILIZE["Phase 4: Stabilize"]
STABILIZE --> CANDIDATES["4a: Find Candidate Fixes"]
CANDIDATES --> CHERRYPICK["4b: Cherry-Pick to Release Branches"]
CHERRYPICK --> NEXTTAG["4c: Tag Next RC"]
NEXTTAG --> VERIFY
STABILIZE -->|"Stable"| GA["Phase 5: Cut GA"]
GA --> NOTES["Phase 6: Release Notes"]
NOTES --> ANNOUNCE["Phase 7: Announce"]
classDef loop fill:#FFF3CD,stroke:#856404,color:#856404
class STABILIZE,CANDIDATES,CHERRYPICK,NEXTTAG loop
Follow this diagram as the workflow. The yellow loop (Phase 4) repeats until
the RC is stable. Reference docs/releasing.md for the full process.
Release Kagenti
Guided workflow for creating and stabilizing releases across the Kagenti
organization. Handles the multi-repo dependency order, RC iteration loop,
cherry-pick coordination, and artifact verification.
Table of Contents
When to Use
- Starting a new release cycle (first alpha or first RC)
- Daily stabilization work between RCs (finding fixes, cherry-picking, tagging)
- Coordinating multi-repo releases across the organization
- Cutting GA after stabilization is complete
- Patching an existing GA release
Invocation
/release # Start from Phase 1 (full assessment)
/release status # Show current release state + fix tracker
/release alpha vX.Y.0-alpha.N # Cut an alpha from main
/release rc vX.Y.0-rc.N # Cut a release candidate
/release stabilize # Enter Phase 4: find fixes, cherry-pick, tag next RC
/release cherry-pick <PR#|SHA> # Cherry-pick a specific fix into the release branch
/release ga vX.Y.0 # Promote to GA
/release patch vX.Y.Z # Cut a patch release
If no arguments are given, start with Phase 1 to assess and ask what to do.
Phase 1: Assess Current State
Gather the current release landscape before making any changes.
1.1 Current tags across repos
echo "=== kagenti/kagenti ==="
gh release list --repo kagenti/kagenti --limit 5
echo "=== kagenti/kagenti-extensions ==="
gh release list --repo kagenti/kagenti-extensions --limit 5
echo "=== kagenti/kagenti-operator ==="
gh release list --repo kagenti/kagenti-operator --limit 5
echo "=== kagenti/agent-examples ==="
gh release list --repo kagenti/agent-examples --limit 5
1.2 Chart dependency versions
grep -A2 'name: kagenti-' charts/kagenti/Chart.yaml
1.3 Image tags in values.yaml
grep -n 'tag:' charts/kagenti/values.yaml charts/kagenti-deps/values.yaml
Flag any tag: latest entries — these must be pinned before any release.
1.4 Release branch state (if exists)
git fetch upstream
git branch -r | grep release
git log --oneline upstream/release-X.Y -5
1.5 Present summary and ask
Present the user with a summary built from live data:
Current state:
kagenti: <latest tag> | release branch: <exists/not>
kagenti-extensions: <latest tag> | release branch: <exists/not>
kagenti-operator: <latest tag> | release branch: <exists/not>
Chart.yaml pins:
kagenti-webhook-chart: <version>
kagenti-operator-chart: <version>
Image tag issues:
<N> images using tag: latest (must fix before RC/GA)
ASK: "What would you like to do? Options:
- Cut a new alpha from main
- Cut the first RC (creates release branch)
- Stabilize an existing RC (cherry-pick fixes, tag next RC)
- Promote to GA
- Cut a patch release"
Phase 2: Prepare and Tag
Dependency order (MANDATORY)
All repos must be tagged in this order. Wait for CI to complete between each:
1. kagenti/kagenti-operator → first
2. kagenti/kagenti-extensions → second
3. kagenti/agent-examples → third (if applicable)
4. kagenti/kagenti → last (update Chart.yaml + values.yaml, then tag)
2.1 For Alpha (from main)
Alphas are tagged directly from main:
bash scripts/pin-release-tags.sh <version>
bash scripts/check-release-pins.sh
2.2 For First RC (creates release branch)
The first RC marks feature freeze. This is when release branches are created.
Prerequisites — ASK the release manager:
- "Are all planned features for vX.Y.0 merged to main?"
- "Are there any open P0/P1 bugs against this milestone?"
- "Has feature freeze been declared on Slack?"
Steps:
-
Tag dependency repos with RC (following dependency order):
For each dependency repo that has changes since the last release:
gh repo clone kagenti/<repo-name> /tmp/kagenti-release/<repo-name>
cd /tmp/kagenti-release/<repo-name>
gh run list --branch main --limit 3
git checkout -b release-X.Y main
git push origin release-X.Y
git tag -s vA.B.0-rc.1 -m "vA.B.0-rc.1"
git push origin vA.B.0-rc.1
gh run watch
ASK after each: "Tag pushed for . CI running. Shall I verify artifacts and proceed to the next repo?"
-
Update kagenti/kagenti Chart.yaml with new sub-chart RC versions
-
Pin image tags and chart version:
bash scripts/pin-release-tags.sh <version>
bash scripts/check-release-pins.sh
-
Create the release branch in kagenti/kagenti:
git checkout main
git pull upstream main
git checkout -b release-X.Y
git push upstream release-X.Y
-
Tag RC1:
git tag -s vX.Y.0-rc.1 -m "vX.Y.0-rc.1"
git push upstream vX.Y.0-rc.1
-
Initialize the fix tracker (see RC Fix Tracking)
After tagging RC1, tell the release manager:
RC1 is tagged. From now on, the stabilization cycle begins:
- Test the RC
- Report bugs → fix them via PRs to
main
- When fixes are merged, run
/release stabilize to cherry-pick them
into the release branch and tag the next RC.
- Repeat until stable, then
/release ga to promote.
2.3 For Subsequent RCs (rc.2, rc.3, ...)
Subsequent RCs are cut from the release branch after cherry-picking fixes.
This is handled by Phase 4: Stabilize.
2.4 Tag a dependency repo (helper)
REPO="kagenti/<repo-name>"
VERSION="<version>"
rm -rf /tmp/kagenti-release/$(basename $REPO)
gh repo clone $REPO /tmp/kagenti-release/$(basename $REPO)
cd /tmp/kagenti-release/$(basename $REPO)
gh run list --branch <main|release-X.Y> --limit 3
git tag -s $VERSION -m "$VERSION"
git push origin $VERSION
gh run watch
After each tag, run verification (see Phase 3) and get explicit user
approval before proceeding to the next repo.
Phase 3: Verify Artifacts
After tagging, verify that CI produced all expected artifacts.
3.1 GitHub Releases
for repo in kagenti kagenti-extensions kagenti-operator; do
echo "=== kagenti/$repo ==="
gh release view <version> --repo kagenti/$repo --json tagName,isPrerelease,publishedAt 2>/dev/null || echo " Not found"
done
3.2 Container images
REGISTRY="ghcr.io/kagenti"
VERSION="<version>"
for img in ui-v2 backend ui-oauth-secret agent-oauth-secret api-oauth-secret; do
echo -n "$img:$VERSION ... "
docker manifest inspect ghcr.io/kagenti/kagenti/$img:$VERSION >/dev/null 2>&1 \
&& echo "OK" || echo "MISSING"
done
for img in envoy-with-processor proxy-init client-registration; do
echo -n "$img:$VERSION ... "
docker manifest inspect ghcr.io/kagenti/kagenti-extensions/$img:$VERSION >/dev/null 2>&1 \
&& echo "OK" || echo "MISSING"
done
3.3 Helm charts
helm show chart oci://ghcr.io/kagenti/kagenti-extensions/kagenti-webhook-chart --version <chart-version> 2>/dev/null \
&& echo "webhook chart OK" || echo "webhook chart MISSING"
helm show chart oci://ghcr.io/kagenti/kagenti-operator/kagenti-operator-chart --version <chart-version> 2>/dev/null \
&& echo "operator chart OK" || echo "operator chart MISSING"
3.4 Pre-release flag
gh release view <version> --repo kagenti/kagenti --json isPrerelease --jq '.isPrerelease'
3.5 E2E validation (optional for RCs, mandatory for GA)
gh workflow run e2e-release-validation.yaml \
-f version=<version> \
--repo kagenti/kagenti
ASK: "All artifacts verified. Is this RC ready for broader testing, or do
you already know of issues to fix?"
Phase 4: Stabilize (RC Iteration Loop)
This is the core loop between RCs. The release manager re-enters this phase
each time fixes need to be incorporated.
Entry point: /release stabilize
4a: Find Candidate Fixes
Discover PRs merged to main since the last RC that may need cherry-picking:
LAST_RC="v0.6.0-rc.6"
LAST_RC_DATE=$(gh release view $LAST_RC --repo kagenti/kagenti --json publishedAt --jq '.publishedAt')
echo "=== PRs merged to kagenti/kagenti main since $LAST_RC ==="
gh pr list --repo kagenti/kagenti --state merged --base main \
--search "merged:>$LAST_RC_DATE" --json number,title,labels,mergeCommit \
--jq '.[] | "#\(.number) \(.title) [\(.mergeCommit.oid[:12])] labels:\([.labels[].name] | join(","))"'
echo ""
echo "=== PRs merged to kagenti/kagenti-extensions main since $LAST_RC ==="
gh pr list --repo kagenti/kagenti-extensions --state merged --base main \
--search "merged:>$LAST_RC_DATE" --json number,title,mergeCommit \
--jq '.[] | "#\(.number) \(.title) [\(.mergeCommit.oid[:12])]"'
echo ""
echo "=== PRs merged to kagenti/kagenti-operator main since $LAST_RC ==="
gh pr list --repo kagenti/kagenti-operator --state merged --base main \
--search "merged:>$LAST_RC_DATE" --json number,title,mergeCommit \
--jq '.[] | "#\(.number) \(.title) [\(.mergeCommit.oid[:12])]"'
Also check the fix tracker for known pending items:
cat /tmp/kagenti/release/<version>/rc-fixes.md 2>/dev/null || echo "No tracker yet — will create one."
ASK the release manager:
These PRs were merged since the last RC. Which should be included in the next RC?
kagenti/kagenti:
1. #1655 - fix(ocp): skip remote tag detection [bafe0d73]
2. #1660 - fix(ui): dashboard crash on empty state [abc123]
3. #1670 - chore(deps): bump go to 1.22 [def456]
kagenti-extensions:
4. #89 - fix(webhook): handle nil annotations [aaa111]
kagenti-operator:
(none)
Select PRs to cherry-pick (comma-separated numbers, e.g. "1,2,4"), or 'none':
4b: Cherry-Pick to Release Branches
For each selected fix, cherry-pick into the appropriate release branch.
Important rules (from SOP):
- All fixes MUST land on
main first — never commit directly to a release branch
- Always use
git cherry-pick -x — the -x flag is mandatory for traceability
- Follow dependency order: operator → extensions → kagenti
If fixes touch dependency repos (extensions or operator)
ASK: "PR #89 is in kagenti-extensions. Does that repo have a release-X.Y
branch yet?"
If no release branch exists for the dependency repo:
gh repo clone kagenti/kagenti-extensions /tmp/kagenti-release/kagenti-extensions
cd /tmp/kagenti-release/kagenti-extensions
git checkout -b release-X.Y main
git push origin release-X.Y
Then cherry-pick and tag:
cd /tmp/kagenti-release/kagenti-extensions
git checkout release-X.Y
git pull origin release-X.Y
git cherry-pick -x <merge-commit-sha>
git push origin release-X.Y
git tag -s vA.B.0-rc.N -m "vA.B.0-rc.N"
git push origin vA.B.0-rc.N
gh run watch
After dependency repo RC is tagged: Update charts/kagenti/Chart.yaml in
the kagenti release branch to reference the new dependency version.
Cherry-pick into kagenti/kagenti release branch
Use the git workflow described in Release Branch Git Workflow:
git fetch upstream release-X.Y
git checkout release-X.Y 2>/dev/null || git checkout -b release-X.Y upstream/release-X.Y
git reset --hard upstream/release-X.Y
git cherry-pick -x <sha1>
git cherry-pick -x <sha2>
git push upstream release-X.Y
Update the fix tracker
After cherry-picking, update the local tracking file (see RC Fix Tracking).
4c: Tag Next RC
Once all cherry-picks are on the release branch(es) and CI passes:
gh run list --repo kagenti/kagenti --branch release-X.Y --limit 3
git tag --list 'vX.Y.0-rc.*' --sort=-v:refname | head -1
Pin image tags and chart version for the new RC:
bash scripts/pin-release-tags.sh <next-rc-version>
bash scripts/check-release-pins.sh
git add charts/
git commit -s -m "chore(release): pin image tags for <next-rc-version>"
git push upstream release-X.Y
ASK: "Release branch has N new commits since rc.N. Ready to tag rc.N+1?"
git tag -s vX.Y.0-rc.N+1 -m "vX.Y.0-rc.N+1"
git push upstream vX.Y.0-rc.N+1
→ Return to Phase 3: Verify Artifacts for the new RC.
Phase 5: Cut GA
When the latest RC has soaked with no blocking issues.
Prerequisites — ASK:
- "How long has it been since the last RC? (recommend 1 week minimum)"
- "Are there any open release-blocking issues?"
- "Has another maintainer signed off on this RC?"
Steps:
-
Tag dependency repos with GA (following dependency order):
cd /tmp/kagenti-release/<repo>
git checkout release-X.Y
git tag -s vA.B.0 -m "vA.B.0"
git push origin vA.B.0
gh run watch
-
Update kagenti Chart.yaml with GA sub-chart versions
-
Pin image tags and chart version to GA:
bash scripts/pin-release-tags.sh vX.Y.0
bash scripts/check-release-pins.sh
git add charts/
git commit -s -m "chore(release): pin image tags for vX.Y.0"
git push upstream release-X.Y
-
Tag GA:
git tag -s vX.Y.0 -m "vX.Y.0"
git push upstream vX.Y.0
-
Verify (Phase 3) — confirm GitHub Release is marked as "Latest" (not Pre-release)
-
Mark the release as "Latest" if needed:
gh release edit vX.Y.0 --repo kagenti/kagenti --latest
Phase 6: Release Notes
For Alpha
Auto-generated is sufficient:
gh release edit <version> --repo kagenti/kagenti \
--notes "Alpha release — known issues: <list any>"
4.5 Run RC Validation CI (RC and GA only)
After verifying artifacts for an RC tag, ask the user:
RC artifacts verified. Would you like to trigger the RC Validation workflow?
This runs Kind + HyperShift E2E tests in parallel (~2 hours).
Options:
1. Run full validation (Kind + HyperShift)
2. Run Kind only (faster, ~90 min)
3. Run HyperShift only (~120 min)
4. Skip — I'll validate manually
If the user chooses to run validation:
gh workflow run release-validation.yaml -f ref=<version>
gh workflow run release-validation.yaml -f ref=<version> -f skip_hypershift=true
gh workflow run release-validation.yaml -f ref=<version> -f skip_kind=true
If dependency repos were also tagged, include them:
gh workflow run release-validation.yaml -f ref=<version> \
-f dep_builds='[{"repo":"kagenti/kagenti-extensions","ref":"<ext-version>"}]'
After triggering, show the run URL:
sleep 5
gh run list --workflow=release-validation.yaml --limit 1 --json databaseId,url --jq '.[0].url'
For GA releases, this validation MUST have passed on the final RC before
tagging GA. Remind the user:
GA prerequisite: Has the RC Validation workflow passed for the latest RC?
Check with: gh run list --workflow=release-validation.yaml --limit 5
For RC
Include a testing checklist and changes since last RC:
gh release edit <version> --repo kagenti/kagenti --notes-file /tmp/rc-notes.md
Template:
Release candidate for vX.Y.0.
## Testing needed
- [ ] Clean Kind install
- [ ] OpenShift install
- [ ] Upgrade from previous GA
- [ ] E2E tests
## Changes since <previous-rc>
<list from fix tracker>
For GA
Full release notes with component compatibility table:
## Highlights
- Feature 1
- Feature 2
## Breaking Changes
- (list any)
## Component Versions
| Component | Version |
|-----------|---------|
| kagenti (platform) | vX.Y.0 |
| kagenti-extensions (webhook) | vA.B.0 |
| kagenti-operator | vC.D.0 |
| agent-examples | vE.F.0 |
## Upgrade Notes
- (special steps from previous GA)
## Full Changelog
<auto-generated>
Phase 7: Announce
For GA and significant RC releases:
echo "Announce on:"
echo " - Slack: https://ibm.biz/kagenti-slack"
echo " - Mailing list: kagenti-maintainers@googlegroups.com"
Release Branch Git Workflow
This section describes how to work with release branches day-to-day.
Core Rule (from SOP)
No direct commits to release branches. All fixes land on main first
and are cherry-picked back. The -x flag is mandatory for traceability.
Approach A: Direct push (maintainers)
Use when you have write access to the upstream repo and the cherry-picks are
straightforward:
git fetch upstream release-X.Y
git checkout release-X.Y 2>/dev/null || git checkout -b release-X.Y upstream/release-X.Y
git reset --hard upstream/release-X.Y
git cherry-pick -x <sha1>
git cherry-pick -x <sha2>
git push upstream release-X.Y
When conflicts occur:
git status
git add <resolved-files>
git cherry-pick --continue
git cherry-pick --abort
Approach B: PR to release branch
Use when you want review on the cherry-pick, don't have direct push access,
or the cherry-pick has non-trivial conflicts:
git fetch upstream release-X.Y
git checkout -b cherry-pick-<desc> upstream/release-X.Y
git cherry-pick -x <sha1>
git push origin cherry-pick-<desc>
gh pr create --base release-X.Y --repo kagenti/kagenti \
--title "fix: cherry-pick <description> for rc.N" \
--body "Cherry-pick of #<original-PR> for the vX.Y.0-rc.N release."
When to use which approach
| Scenario | Approach |
|---|
| Release manager with upstream push access, clean cherry-picks | A (direct push) |
| Cherry-pick has conflicts that need a second pair of eyes | B (PR) |
| Contributor without upstream write access | B (PR) |
| Large or risky changes being backported | B (PR) |
| Quick fix already reviewed on the main PR | A (direct push) |
Multi-repo cherry-pick order
When fixes span multiple repos, cherry-pick and tag in dependency order:
1. kagenti-operator (if affected) → cherry-pick, tag RC
2. kagenti-extensions (if affected) → cherry-pick, tag RC
3. kagenti/kagenti → update Chart.yaml deps, cherry-pick fixes, tag RC
RC Fix Tracking
Between RCs, maintain a local tracking file to keep state across sessions.
File location
/tmp/kagenti/release/<version>/rc-fixes.md
Example: /tmp/kagenti/release/v0.6.0/rc-fixes.md
Create the tracker
mkdir -p /tmp/kagenti/release/v0.6.0
cat > /tmp/kagenti/release/v0.6.0/rc-fixes.md << 'EOF'
<!-- - [x] PR
<!-- - [ ] PR
<!-- - [ ] PR
<!-- - [ ] kagenti-extensions PR
<!-- - [ ] kagenti-operator PR
- Initial release candidate
EOF
Update the tracker
After each stabilization cycle:
- Move cherry-picked items to "Cherry-picked" with
[x]
- Add newly discovered candidates to "Pending"
- After tagging, move the "Fixes for next RC" section to "Previous RCs"
Use the tracker for release notes
The tracker provides the changelog between RCs:
grep '^\- \[x\]' /tmp/kagenti/release/v0.6.0/rc-fixes.md
Quick Reference
Release types
| Type | Branch | Create release branch? | Stabilization loop? |
|---|
| Alpha | main | No | No |
| RC (first) | release-X.Y (new) | Yes — from main | Yes — loop until stable |
| RC (subsequent) | release-X.Y | Already exists | Yes — continue loop |
| GA | release-X.Y | Already exists | No — promote last RC |
| Patch | release-X.Y | Already exists | Optional (for non-trivial) |
The stabilization cycle (daily workflow)
1. Test current RC
2. Report bugs → create PRs targeting main
3. Once fixes merge to main:
/release stabilize
→ discovers candidate fixes
→ cherry-picks to release branch(es)
→ tags next RC
4. Verify new RC artifacts
5. Repeat from step 1
Mandatory flags and conventions
| Rule | Reason |
|---|
git cherry-pick -x | Traceability between main and release branch |
git tag -s (or -a if no GPG) | Signed/annotated tags for audit |
git commit -s | DCO sign-off required by CI |
| Pin all image tags before any release | Reproducible installs |
Related Skills
docs/releasing.md — Full release process and policy
git:commit — Commit format conventions
ci:status — CI failure analysis
github:pr-review — PR review workflow