with one click
release
// Full release lifecycle for Kagenti — alpha, RC iteration loop, GA, and patch releases with multi-repo coordination
// Full release lifecycle for Kagenti — alpha, RC iteration loop, GA, and patch releases with multi-repo coordination
TDD iteration loop across 4 environments (local Kind, custom HyperShift, CI Kind, CI HyperShift) with test matrix tracking and log analysis
Create and manage git worktrees for parallel development and testing
Deploy and manage Kagenti operator, agents, and tools on Kubernetes. Handles installer, CRDs, pipelines, and demo deployments.
Deploy the weather agent and MCP tool demo via CLI (no UI required). Uses pre-built ghcr.io images, deploys to team1. Optimized for speed (~15s).
Manage Kind clusters for local Kagenti testing. Create, destroy, deploy platform, and run E2E tests.
Run full end-to-end test workflows for Kagenti. Supports Kind and HyperShift clusters with automated setup and testing.
| name | release |
| description | Full release lifecycle for Kagenti — alpha, RC iteration loop, GA, and patch releases with multi-repo coordination |
flowchart TD
START(["/release"]) --> ASSESS["Phase 1: Assess"]
ASSESS --> PREPARE["Phase 2: Prepare & Tag"]
PREPARE --> VERIFY["Phase 3: Verify Artifacts"]
VERIFY --> STABILIZE["Phase 4: Stabilize"]
STABILIZE --> CANDIDATES["4a: Find Candidate Fixes"]
CANDIDATES --> CHERRYPICK["4b: Cherry-Pick to Release Branches"]
CHERRYPICK --> NEXTTAG["4c: Tag Next RC"]
NEXTTAG --> VERIFY
STABILIZE -->|"Stable"| GA["Phase 5: Cut GA"]
GA --> NOTES["Phase 6: Release Notes"]
NOTES --> ANNOUNCE["Phase 7: Announce"]
classDef loop fill:#FFF3CD,stroke:#856404,color:#856404
class STABILIZE,CANDIDATES,CHERRYPICK,NEXTTAG loop
Follow this diagram as the workflow. The yellow loop (Phase 4) repeats until the RC is stable. Reference
docs/releasing.mdfor the full process.
Guided workflow for creating and stabilizing releases across the Kagenti organization. Handles the multi-repo dependency order, RC iteration loop, cherry-pick coordination, and artifact verification.
/release # Start from Phase 1 (full assessment)
/release status # Show current release state + fix tracker
/release alpha vX.Y.0-alpha.N # Cut an alpha from main
/release rc vX.Y.0-rc.N # Cut a release candidate
/release stabilize # Enter Phase 4: find fixes, cherry-pick, tag next RC
/release cherry-pick <PR#|SHA> # Cherry-pick a specific fix into the release branch
/release ga vX.Y.0 # Promote to GA
/release patch vX.Y.Z # Cut a patch release
If no arguments are given, start with Phase 1 to assess and ask what to do.
Gather the current release landscape before making any changes.
echo "=== kagenti/kagenti ==="
gh release list --repo kagenti/kagenti --limit 5
echo "=== kagenti/kagenti-extensions ==="
gh release list --repo kagenti/kagenti-extensions --limit 5
echo "=== kagenti/kagenti-operator ==="
gh release list --repo kagenti/kagenti-operator --limit 5
echo "=== kagenti/agent-examples ==="
gh release list --repo kagenti/agent-examples --limit 5
grep -A2 'name: kagenti-' charts/kagenti/Chart.yaml
grep -n 'tag:' charts/kagenti/values.yaml charts/kagenti-deps/values.yaml
Flag any tag: latest entries — these must be pinned before any release.
git fetch upstream
git branch -r | grep release
# For each release branch:
git log --oneline upstream/release-X.Y -5
Present the user with a summary built from live data:
Current state:
kagenti: <latest tag> | release branch: <exists/not>
kagenti-extensions: <latest tag> | release branch: <exists/not>
kagenti-operator: <latest tag> | release branch: <exists/not>
Chart.yaml pins:
kagenti-webhook-chart: <version>
kagenti-operator-chart: <version>
Image tag issues:
<N> images using tag: latest (must fix before RC/GA)
ASK: "What would you like to do? Options:
All repos must be tagged in this order. Wait for CI to complete between each:
1. kagenti/kagenti-operator → first
2. kagenti/kagenti-extensions → second
3. kagenti/agent-examples → third (if applicable)
4. kagenti/kagenti → last (update Chart.yaml + values.yaml, then tag)
main)Alphas are tagged directly from main:
main for each repo being taggedtag: latest allowed, even for alphas):bash scripts/pin-release-tags.sh <version>
bash scripts/check-release-pins.sh
Tag dependency repos first (operator → extensions → agent-examples)
Update charts/kagenti/Chart.yaml with new sub-chart versions
Run helm dependency update charts/kagenti/
Commit, merge to main, then tag kagenti/kagenti
At least one RC validated via release-validation.yaml (Kind + HyperShift pass)
Minimum 1 week soak since last RC (recommended)
No open release-blocking issues
At least one maintainer sign-off
Dependency repos tagged with GA first
charts/kagenti/Chart.yaml updated with GA sub-chart versions
helm dependency update charts/kagenti/ regenerates Chart.lock
All image tags in charts/kagenti/values.yaml pinned to GA tag
Documentation reviewed and updated
The first RC marks feature freeze. This is when release branches are created.
Prerequisites — ASK the release manager:
Steps:
Tag dependency repos with RC (following dependency order):
For each dependency repo that has changes since the last release:
gh repo clone kagenti/<repo-name> /tmp/kagenti-release/<repo-name>
cd /tmp/kagenti-release/<repo-name>
# Verify CI
gh run list --branch main --limit 3
# Create release branch
git checkout -b release-X.Y main
git push origin release-X.Y
# Tag from the release branch
git tag -s vA.B.0-rc.1 -m "vA.B.0-rc.1"
git push origin vA.B.0-rc.1
# Wait for CI
gh run watch
ASK after each: "Tag pushed for . CI running. Shall I verify artifacts and proceed to the next repo?"
Update kagenti/kagenti Chart.yaml with new sub-chart RC versions
Pin all image tags:
bash scripts/pin-release-tags.sh <version>
bash scripts/check-release-pins.sh
Create the release branch in kagenti/kagenti:
git checkout main
git pull upstream main
git checkout -b release-X.Y
git push upstream release-X.Y
Tag RC1:
git tag -s vX.Y.0-rc.1 -m "vX.Y.0-rc.1"
git push upstream vX.Y.0-rc.1
Initialize the fix tracker (see RC Fix Tracking)
After tagging RC1, tell the release manager:
RC1 is tagged. From now on, the stabilization cycle begins:
- Test the RC
- Report bugs → fix them via PRs to
main- When fixes are merged, run
/release stabilizeto cherry-pick them into the release branch and tag the next RC.- Repeat until stable, then
/release gato promote.
Subsequent RCs are cut from the release branch after cherry-picking fixes. This is handled by Phase 4: Stabilize.
REPO="kagenti/<repo-name>"
VERSION="<version>"
# Clone clean
rm -rf /tmp/kagenti-release/$(basename $REPO)
gh repo clone $REPO /tmp/kagenti-release/$(basename $REPO)
cd /tmp/kagenti-release/$(basename $REPO)
# Verify CI on the target branch
gh run list --branch <main|release-X.Y> --limit 3
# Create signed tag
git tag -s $VERSION -m "$VERSION"
git push origin $VERSION
# Wait and verify
gh run watch
After each tag, run verification (see Phase 3) and get explicit user approval before proceeding to the next repo.
After tagging, verify that CI produced all expected artifacts.
for repo in kagenti kagenti-extensions kagenti-operator; do
echo "=== kagenti/$repo ==="
gh release view <version> --repo kagenti/$repo --json tagName,isPrerelease,publishedAt 2>/dev/null || echo " Not found"
done
REGISTRY="ghcr.io/kagenti"
VERSION="<version>"
# kagenti/kagenti images
for img in ui-v2 backend ui-oauth-secret agent-oauth-secret api-oauth-secret; do
echo -n "$img:$VERSION ... "
docker manifest inspect ghcr.io/kagenti/kagenti/$img:$VERSION >/dev/null 2>&1 \
&& echo "OK" || echo "MISSING"
done
# kagenti-extensions images
for img in envoy-with-processor proxy-init client-registration; do
echo -n "$img:$VERSION ... "
docker manifest inspect ghcr.io/kagenti/kagenti-extensions/$img:$VERSION >/dev/null 2>&1 \
&& echo "OK" || echo "MISSING"
done
helm show chart oci://ghcr.io/kagenti/kagenti-extensions/kagenti-webhook-chart --version <chart-version> 2>/dev/null \
&& echo "webhook chart OK" || echo "webhook chart MISSING"
helm show chart oci://ghcr.io/kagenti/kagenti-operator/kagenti-operator-chart --version <chart-version> 2>/dev/null \
&& echo "operator chart OK" || echo "operator chart MISSING"
gh release view <version> --repo kagenti/kagenti --json isPrerelease --jq '.isPrerelease'
# Expected: true for alpha/RC, false for GA
gh workflow run e2e-release-validation.yaml \
-f version=<version> \
--repo kagenti/kagenti
ASK: "All artifacts verified. Is this RC ready for broader testing, or do you already know of issues to fix?"
This is the core loop between RCs. The release manager re-enters this phase each time fixes need to be incorporated.
Entry point: /release stabilize
Discover PRs merged to main since the last RC that may need cherry-picking:
# Get the date of the last RC
LAST_RC="v0.6.0-rc.6" # adjust to actual
LAST_RC_DATE=$(gh release view $LAST_RC --repo kagenti/kagenti --json publishedAt --jq '.publishedAt')
echo "=== PRs merged to kagenti/kagenti main since $LAST_RC ==="
gh pr list --repo kagenti/kagenti --state merged --base main \
--search "merged:>$LAST_RC_DATE" --json number,title,labels,mergeCommit \
--jq '.[] | "#\(.number) \(.title) [\(.mergeCommit.oid[:12])] labels:\([.labels[].name] | join(","))"'
echo ""
echo "=== PRs merged to kagenti/kagenti-extensions main since $LAST_RC ==="
gh pr list --repo kagenti/kagenti-extensions --state merged --base main \
--search "merged:>$LAST_RC_DATE" --json number,title,mergeCommit \
--jq '.[] | "#\(.number) \(.title) [\(.mergeCommit.oid[:12])]"'
echo ""
echo "=== PRs merged to kagenti/kagenti-operator main since $LAST_RC ==="
gh pr list --repo kagenti/kagenti-operator --state merged --base main \
--search "merged:>$LAST_RC_DATE" --json number,title,mergeCommit \
--jq '.[] | "#\(.number) \(.title) [\(.mergeCommit.oid[:12])]"'
Also check the fix tracker for known pending items:
cat /tmp/kagenti/release/<version>/rc-fixes.md 2>/dev/null || echo "No tracker yet — will create one."
ASK the release manager:
These PRs were merged since the last RC. Which should be included in the next RC?
kagenti/kagenti:
1. #1655 - fix(ocp): skip remote tag detection [bafe0d73]
2. #1660 - fix(ui): dashboard crash on empty state [abc123]
3. #1670 - chore(deps): bump go to 1.22 [def456]
kagenti-extensions:
4. #89 - fix(webhook): handle nil annotations [aaa111]
kagenti-operator:
(none)
Select PRs to cherry-pick (comma-separated numbers, e.g. "1,2,4"), or 'none':
For each selected fix, cherry-pick into the appropriate release branch.
Important rules (from SOP):
main first — never commit directly to a release branchgit cherry-pick -x — the -x flag is mandatory for traceabilityASK: "PR #89 is in kagenti-extensions. Does that repo have a release-X.Y branch yet?"
If no release branch exists for the dependency repo:
# Create release branch in the dependency repo
gh repo clone kagenti/kagenti-extensions /tmp/kagenti-release/kagenti-extensions
cd /tmp/kagenti-release/kagenti-extensions
git checkout -b release-X.Y main
git push origin release-X.Y
Then cherry-pick and tag:
cd /tmp/kagenti-release/kagenti-extensions
git checkout release-X.Y
git pull origin release-X.Y
git cherry-pick -x <merge-commit-sha>
git push origin release-X.Y
# Tag the dependency repo RC
git tag -s vA.B.0-rc.N -m "vA.B.0-rc.N"
git push origin vA.B.0-rc.N
gh run watch
After dependency repo RC is tagged: Update charts/kagenti/Chart.yaml in
the kagenti release branch to reference the new dependency version.
Use the git workflow described in Release Branch Git Workflow:
# Sync local release branch with upstream
git fetch upstream release-X.Y
git checkout release-X.Y 2>/dev/null || git checkout -b release-X.Y upstream/release-X.Y
git reset --hard upstream/release-X.Y
# Cherry-pick each fix (use merge commit SHA for squash-merged PRs)
git cherry-pick -x <sha1>
git cherry-pick -x <sha2>
# If a cherry-pick has conflicts:
# ASK: "Cherry-pick of <sha> conflicts in <files>. Options:
# 1. I'll resolve manually (show me the conflict)
# 2. Skip this commit for now
# 3. Abort and investigate"
# Push to upstream
git push upstream release-X.Y
After cherry-picking, update the local tracking file (see RC Fix Tracking).
Once all cherry-picks are on the release branch(es) and CI passes:
# Verify CI on release branch
gh run list --repo kagenti/kagenti --branch release-X.Y --limit 3
# Determine next RC number
git tag --list 'vX.Y.0-rc.*' --sort=-v:refname | head -1
Pin image tags for the new RC:
bash scripts/pin-release-tags.sh <next-rc-version>
bash scripts/check-release-pins.sh
git add charts/
git commit -s -m "chore(release): pin image tags for <next-rc-version>"
git push upstream release-X.Y
ASK: "Release branch has N new commits since rc.N. Ready to tag rc.N+1?"
git tag -s vX.Y.0-rc.N+1 -m "vX.Y.0-rc.N+1"
git push upstream vX.Y.0-rc.N+1
→ Return to Phase 3: Verify Artifacts for the new RC.
When the latest RC has soaked with no blocking issues.
Prerequisites — ASK:
Steps:
Tag dependency repos with GA (following dependency order):
# For each dependency repo that had RC tags
cd /tmp/kagenti-release/<repo>
git checkout release-X.Y
git tag -s vA.B.0 -m "vA.B.0"
git push origin vA.B.0
gh run watch
Update kagenti Chart.yaml with GA sub-chart versions
Pin image tags to GA version:
bash scripts/pin-release-tags.sh vX.Y.0
bash scripts/check-release-pins.sh
git add charts/
git commit -s -m "chore(release): pin image tags for vX.Y.0"
git push upstream release-X.Y
Tag GA:
git tag -s vX.Y.0 -m "vX.Y.0"
git push upstream vX.Y.0
Verify (Phase 3) — confirm GitHub Release is marked as "Latest" (not Pre-release)
Mark the release as "Latest" if needed:
gh release edit vX.Y.0 --repo kagenti/kagenti --latest
Auto-generated is sufficient:
gh release edit <version> --repo kagenti/kagenti \
--notes "Alpha release — known issues: <list any>"
After verifying artifacts for an RC tag, ask the user:
RC artifacts verified. Would you like to trigger the RC Validation workflow?
This runs Kind + HyperShift E2E tests in parallel (~2 hours).
Options:
1. Run full validation (Kind + HyperShift)
2. Run Kind only (faster, ~90 min)
3. Run HyperShift only (~120 min)
4. Skip — I'll validate manually
If the user chooses to run validation:
# Full validation (default)
gh workflow run release-validation.yaml -f ref=<version>
# Kind only
gh workflow run release-validation.yaml -f ref=<version> -f skip_hypershift=true
# HyperShift only
gh workflow run release-validation.yaml -f ref=<version> -f skip_kind=true
If dependency repos were also tagged, include them:
gh workflow run release-validation.yaml -f ref=<version> \
-f dep_builds='[{"repo":"kagenti/kagenti-extensions","ref":"<ext-version>"}]'
After triggering, show the run URL:
sleep 5
gh run list --workflow=release-validation.yaml --limit 1 --json databaseId,url --jq '.[0].url'
For GA releases, this validation MUST have passed on the final RC before tagging GA. Remind the user:
GA prerequisite: Has the RC Validation workflow passed for the latest RC?
Check with: gh run list --workflow=release-validation.yaml --limit 5
Include a testing checklist and changes since last RC:
gh release edit <version> --repo kagenti/kagenti --notes-file /tmp/rc-notes.md
Template:
Release candidate for vX.Y.0.
## Testing needed
- [ ] Clean Kind install
- [ ] OpenShift install
- [ ] Upgrade from previous GA
- [ ] E2E tests
## Changes since <previous-rc>
<list from fix tracker>
Full release notes with component compatibility table:
## Highlights
- Feature 1
- Feature 2
## Breaking Changes
- (list any)
## Component Versions
| Component | Version |
|-----------|---------|
| kagenti (platform) | vX.Y.0 |
| kagenti-extensions (webhook) | vA.B.0 |
| kagenti-operator | vC.D.0 |
| agent-examples | vE.F.0 |
## Upgrade Notes
- (special steps from previous GA)
## Full Changelog
<auto-generated>
For GA and significant RC releases:
echo "Announce on:"
echo " - Slack: https://ibm.biz/kagenti-slack"
echo " - Mailing list: kagenti-maintainers@googlegroups.com"
This section describes how to work with release branches day-to-day.
No direct commits to release branches. All fixes land on
mainfirst and are cherry-picked back. The-xflag is mandatory for traceability.
Use when you have write access to the upstream repo and the cherry-picks are straightforward:
# 1. Sync local branch with upstream
git fetch upstream release-X.Y
git checkout release-X.Y 2>/dev/null || git checkout -b release-X.Y upstream/release-X.Y
git reset --hard upstream/release-X.Y
# 2. Cherry-pick (use -x for traceability)
git cherry-pick -x <sha1>
git cherry-pick -x <sha2>
# For squash-merged PRs, use the merge commit SHA shown in the PR
# 3. Push directly to upstream
git push upstream release-X.Y
When conflicts occur:
# After a conflicting cherry-pick:
git status # See conflicted files
# ... resolve conflicts ...
git add <resolved-files>
git cherry-pick --continue
# If the conflict is too complex, abort and try a different approach:
git cherry-pick --abort
Use when you want review on the cherry-pick, don't have direct push access, or the cherry-pick has non-trivial conflicts:
# 1. Create a branch from the release branch
git fetch upstream release-X.Y
git checkout -b cherry-pick-<desc> upstream/release-X.Y
# 2. Cherry-pick
git cherry-pick -x <sha1>
# 3. Push to your fork
git push origin cherry-pick-<desc>
# 4. Open PR targeting the release branch
gh pr create --base release-X.Y --repo kagenti/kagenti \
--title "fix: cherry-pick <description> for rc.N" \
--body "Cherry-pick of #<original-PR> for the vX.Y.0-rc.N release."
| Scenario | Approach |
|---|---|
| Release manager with upstream push access, clean cherry-picks | A (direct push) |
| Cherry-pick has conflicts that need a second pair of eyes | B (PR) |
| Contributor without upstream write access | B (PR) |
| Large or risky changes being backported | B (PR) |
| Quick fix already reviewed on the main PR | A (direct push) |
When fixes span multiple repos, cherry-pick and tag in dependency order:
1. kagenti-operator (if affected) → cherry-pick, tag RC
2. kagenti-extensions (if affected) → cherry-pick, tag RC
3. kagenti/kagenti → update Chart.yaml deps, cherry-pick fixes, tag RC
Between RCs, maintain a local tracking file to keep state across sessions.
/tmp/kagenti/release/<version>/rc-fixes.md
Example: /tmp/kagenti/release/v0.6.0/rc-fixes.md
mkdir -p /tmp/kagenti/release/v0.6.0
cat > /tmp/kagenti/release/v0.6.0/rc-fixes.md << 'EOF'
# v0.6.0 RC Fix Tracker
## Current RC: rc.1 (tagged YYYY-MM-DD)
## Fixes for next RC
### Cherry-picked (ready to tag)
<!-- - [x] PR #NNN - description (commits: sha1, sha2) -->
### Pending (merged to main, not yet cherry-picked)
<!-- - [ ] PR #NNN - description [merge-sha] -->
### In Progress (PR open targeting main, not yet merged)
<!-- - [ ] PR #NNN - description -->
### Dependency repo fixes
<!-- - [ ] kagenti-extensions PR #NN - description -->
<!-- - [ ] kagenti-operator PR #NN - description -->
## Previous RCs
### rc.1 (initial)
- Initial release candidate
EOF
After each stabilization cycle:
[x]The tracker provides the changelog between RCs:
grep '^\- \[x\]' /tmp/kagenti/release/v0.6.0/rc-fixes.md
| Type | Branch | Create release branch? | Stabilization loop? |
|---|---|---|---|
| Alpha | main | No | No |
| RC (first) | release-X.Y (new) | Yes — from main | Yes — loop until stable |
| RC (subsequent) | release-X.Y | Already exists | Yes — continue loop |
| GA | release-X.Y | Already exists | No — promote last RC |
| Patch | release-X.Y | Already exists | Optional (for non-trivial) |
1. Test current RC
2. Report bugs → create PRs targeting main
3. Once fixes merge to main:
/release stabilize
→ discovers candidate fixes
→ cherry-picks to release branch(es)
→ tags next RC
4. Verify new RC artifacts
5. Repeat from step 1
| Rule | Reason |
|---|---|
git cherry-pick -x | Traceability between main and release branch |
git tag -s (or -a if no GPG) | Signed/annotated tags for audit |
git commit -s | DCO sign-off required by CI |
| Pin all image tags before any release | Reproducible installs |
docs/releasing.md — Full release process and policygit:commit — Commit format conventionsci:status — CI failure analysisgithub:pr-review — PR review workflow