release

name	release
description	Verify and promote published artifacts safely. Use when checking digests, signatures, stable releases, rollback, or emergency promotion.
metadata	{"context7-sources":["/podman-container-tools/skopeo"]}

Release

When to Use

• Editing .github/workflows/promote-testing-to-main.yml or execute-release.yml
• Debugging why a promotion PR did not auto-merge
• Performing an emergency production rollback
• Verifying published :stable images after a release

When NOT to Use

• Package or image-content changes → docs/skills/build/SKILL.md
• Non-promotion CI failures → docs/skills/ci-cd/SKILL.md
• Hardware-specific image issues → docs/skills/hardware/SKILL.md

Branch model (factory standard)

testing → main → :stable

• testing — all PRs target this branch. Builds push :testing OCI tag directly on every push.
• main — production source. Advances only via squash promotion from testing. Triggering execute-release.yml.
• No lts branch in the promotion flow. The lts git branch is archived.

Never push directly to main. All changes via PR to testing, then auto-promoted. Flow is one-way: testing → main. Never merge main → testing manually.

Production release flow

1. promote-testing-to-main.yml fires on push to testing, daily at 04:00 UTC, and on manual dispatch. It calls reusable-promote-squash.yml@v1 with explicit source_branch: testing and target_branch: main, and enables use_merge_queue: true. When trees differ it rebuilds the squash branch and upserts the auto/promote-testing-to-main PR.
2. The PR enters the merge queue (ruleset 17070416 on main requires squash + merge queue). Required check: Lint & syntax. No approvals needed.
3. On merge, execute-release.yml fires on push: main, detects the commit message "^chore: promote testing to main", skopeo-copies :testing → :stable for both image variants, and creates a GitHub release with changelog via reusable-release.yml@v1. Stable promotion writes an OCI index, verifies its amd64/arm64 child digests, then signs and verifies the promoted digest because registry media-type conversion can change the index digest.

# Check promotion PR status
gh pr list --repo projectbluefin/bluefin-lts --head auto/promote-testing-to-main

# Emergency bypass (CODEOWNERS or merge queue blocking — requires admin)
gh pr merge <pr-number> --repo projectbluefin/bluefin-lts --squash --admin

Branch protection

main requires a PR with merge queue entry. Gate check: Lint & syntax. 0 approvals required. .github/workflows/ is CODEOWNERS-protected — PRs touching workflow files require --admin bypass.

Daily cadence

• Fully automated: daily 04:00 UTC — cron fires, promote workflow updates the PR, merge queue processes it, execute-release publishes :stable
• No human approval required — Lint & syntax is the only gate
• workflow_dispatch is the supported manual release path

Image verification — always check digests

Do NOT trust "the fix is in main" as evidence the fix is published. Verify:

GHCR_TOKEN=$(gh auth token)
for IMAGE in bluefin-lts bluefin-lts-nvidia; do
  skopeo inspect --no-tags --creds "castrojo:${GHCR_TOKEN}" docker://ghcr.io/projectbluefin/${IMAGE}:stable \
    | python3 -c "import json,sys; d=json.load(sys.stdin); print('${IMAGE}:stable', d['Digest'][:22], d['Labels'].get('org.opencontainers.image.created','?')[:10])"
done

A fix is published when:

1. The :stable digest differs from the last known digest
2. The org.opencontainers.image.created date is after the fix merged
3. Both variants (bluefin-lts and bluefin-lts-nvidia) are updated
4. skopeo inspect --raw shows a complete amd64/arm64 index when ARM artifacts are available; an NVIDIA amd64-only fallback is accepted only when its standalone ARM workflow has failed and the promoted artifact is verified as amd64.

Build cascade — rapid commits cancel in-progress builds

Each push to testing triggers new builds which cancel in-progress builds via concurrency groups. Builds take 60-90 min. Stop committing to testing while builds are in progress.

SHA=<commit-sha>
gh run list --repo projectbluefin/bluefin-lts \
  --json workflowName,status,conclusion,headSha \
  --jq "[.[] | select(.headSha | startswith(\"$SHA\")) | select(.workflowName | contains(\"Build\"))]"

Registry queries

gh auth token | skopeo login ghcr.io -u castrojo --password-stdin
skopeo list-tags docker://ghcr.io/projectbluefin/bluefin-lts
skopeo list-tags docker://ghcr.io/projectbluefin/bluefin-lts-nvidia

Images publish to:

• ghcr.io/projectbluefin/bluefin-lts
• ghcr.io/projectbluefin/bluefin-lts-nvidia

Emergency rollback

GHCR_TOKEN=$(gh auth token)
skopeo copy \
  --src-no-creds \
  --dest-creds "castrojo:${GHCR_TOKEN}" \
  docker://ghcr.io/projectbluefin/IMAGE:stable-YYYYMMDD \
  docker://ghcr.io/projectbluefin/IMAGE:stable

Rollback both variants, then verify digest/created time.

Emergency promotion for production-bricking bugs

1. Push fix to testing — builds trigger automatically on push to testing.
2. Wait for both builds to complete (~45-90 min). Never promote before builds finish.
3. Verify the new :testing image has a fresh initramfs (see Verifying images below).
4. Skopeo-copy :testing → :stable by digest:

GHCR_TOKEN=$(gh auth token)
for IMAGE in bluefin-lts bluefin-lts-nvidia; do
  DIGEST=$(skopeo inspect --no-tags --creds "castrojo:${GHCR_TOKEN}" docker://ghcr.io/projectbluefin/${IMAGE}:testing \
    | python3 -c "import json,sys; print(json.load(sys.stdin)['Digest'])")
  skopeo copy \
    --src-creds "castrojo:${GHCR_TOKEN}" \
    --dest-creds "castrojo:${GHCR_TOKEN}" \
    docker://ghcr.io/projectbluefin/${IMAGE}@${DIGEST} \
    docker://ghcr.io/projectbluefin/${IMAGE}:stable
done

Always copy by digest, not tag — prevents races with concurrent pushes.

Verifying images

:testing is published directly by the build

Build workflows push per-architecture aliases, then publish one signed :testing index after both architecture builds finish. Post-testing verification receives and checks the exact manifest digest. No stable promotion should proceed from a single-architecture stream tag. PR builds validate that the image builds but do not push to GHCR.

/boot/ is intentionally empty in the OCI image

bootc stores the kernel and initramfs under /usr/lib/modules/<kver>/, not /boot/. An empty /boot/ in the container layer is expected and correct.

podman run --rm ghcr.io/projectbluefin/bluefin-lts:stable bash -c '
  sha256sum /usr/lib/modules/*/initramfs.img
  ls -la /usr/lib/modules/*/vmlinuz
  grep BUILD_ID /etc/os-release
'

Initramfs must differ from the previous broken build

After a dracut-related fix, verify the initramfs SHA changed:

OLD=$(podman run --rm ghcr.io/projectbluefin/bluefin-lts:stable bash -c 'sha256sum /usr/lib/modules/*/initramfs.img' 2>/dev/null | awk '{print $1}')
podman pull ghcr.io/projectbluefin/bluefin-lts:stable
NEW=$(podman run --rm ghcr.io/projectbluefin/bluefin-lts:stable bash -c 'sha256sum /usr/lib/modules/*/initramfs.img' 2>/dev/null | awk '{print $1}')
[ "$OLD" != "$NEW" ] && echo "initramfs changed" || echo "same initramfs — promotion may be a no-op"

ISO status

LTS ISO is disabled. Do not re-enable. Anaconda is broken on CentOS Stream base.

Common Rationalizations

• “The tag is enough.” Record and verify the immutable digest.
• “A green build means a release is trustworthy.” Check signatures and publication results.

Red Flags

• use_merge_queue: false — main requires a merge queue (ruleset 17070416); always use true
• source_branch: main or target_branch: lts — model changed; use testing → main
• adding run_e2e: true — no post-merge-e2e gate; builds publish :testing directly
• describing the schedule as weekly — cadence is daily at 04:00 UTC (0 4 * * *)
• Claiming completion without live verification: Never claim a build-fixing task is "fully complete" without noting that the fix is still pending live verification by the active CI pipeline (which takes 45–90 mins). Always clearly differentiate between local code-level/syntax validation and live OCI container build execution.

Verification

• promote-testing-to-main.yml schedules daily at 0 4 * * *
• use_merge_queue: true
• The promotion caller explicitly sets source_branch: testing and target_branch: main
• execute-release.yml fires on push: main, detects "^chore: promote testing to main", publishes :stable
• Build workflows push :testing on push to testing branch