demo-local-rollout

# Local Image Rollout Use this skill for unpublished `sha-<git-sha>` test tags. Formal releases use semver and `$release-cut-and-demo-roll`. ## Choose the cluster first Do this before Harbor auth, Bazel, signing, or any cluster write. | | **carverauto `demo`** | **farm01** | |---|---|---| | When | User says demo, carverauto, or does not name a cluster | User says farm01, farm cluster, or is already on the farm01 kubeconfig | | Kubeconfig | default / carverauto control plane | `export KUBECONFIG="$HOME/.kube/farm01.yaml"` | | Namespace | `demo` | `serviceradar` | | How it rolls | Argo app `serviceradar-demo-prod` | `helm upgrade --reuse-values` | | Admission | Kyverno: images **must** be signed with OpenBao `cosign-release` | **No Kyverno.** Do **not** OpenBao-sign or port-forward the signer | | Registry | Harbor `registry.carverauto.dev/serviceradar` | Same Harbor project (public; no pull secret) | | Live tag source | Deployments in `demo`, plus git override on `demo/prod-release` | `helm get values serviceradar -n serviceradar` (`global.imageTag`) | Never apply the OpenBao / Kyverno / Argo path to farm01. Never helm-upgrade farm01 thinking it is demo. If only `elixir/web-ng/**` changed **and** the target is `demo`, prefer `$demo-web-ng-fastpath`. That fast path is demo-only. ## Shared workflow 1. Work from the checkout that has the commits under test (often a git worktree). 2. Symlink gitignored Bazel rc files if this is not the primary clone (see below). 3. Tag: `sha-$(git rev-parse HEAD)` (full 40-char SHA; that is what `--stamp` emits). 4. Read the **currently deployed** tag on the chosen cluster before changing anything. 5. Map `git diff --name-only <deployed>..HEAD` to images. Rebuild only what changed. 6. Harbor auth: `./buildbuddy_setup_docker_auth.sh` 7. Build and push changed images. Copy unchanged images from the live tag to the new tag. 8. **Then** follow the cluster-specific roll section. Stop after that section; do not run the other cluster's roll. Do not cut a release, edit `VERSION`, or create git tags. Do not roll `demo-staging`, production, or Docker Compose unless the user changes scope. Never push to `staging`. ## Worktree Bazel remotes `.bazelrc` try-imports `%workspace%/.bazelrc.remote` and `.bazelrc.local`. Both are gitignored (BuildBuddy API key). `git worktree add` does not copy them. Without them, `--config=remote_base` / `--config=ci` fails with `PERMISSION_DENIED: Missing API key`. From the primary clone that already has the files: ```bash PRIMARY=/Users/mfreeman/src/serviceradar WT="$(pwd)" ln -sfn "$PRIMARY/.bazelrc.remote" "$WT/.bazelrc.remote" test -e "$PRIMARY/.bazelrc.local" && ln -sfn "$PRIMARY/.bazelrc.local" "$WT/.bazelrc.local" test -f "$WT/.bazelrc.remote" ``` ## Changed image selection ```bash git diff --name-only <currently-deployed-sha-or-tag>..HEAD ``` - `go/cmd/agent/**`, `go/pkg/agent/**`, `go/pkg/mtr/**` -> `serviceradar-agent` - `elixir/serviceradar_agent_gateway/**`, shared agent control/proto -> `serviceradar-agent-gateway` - `elixir/serviceradar_core/**`, `rust/srql/**` (NIF) -> `serviceradar-core-elx` (and usually `serviceradar-web-ng`) - `elixir/web-ng/**` -> `serviceradar-web-ng` - `proto/**` -> every image that consumes the changed generated code When in doubt, rebuild slightly too much. Phoenix / mix.lock / `MODULE.bazel` / `rules_elixir` changes mean rebuild every Elixir image you will roll. ## Images that use `global.imageTag` Copy or rebuild every first-party image the target cluster will pull at the new tag. Inspect live deployments; do not copy demo-only images onto farm01. Typical **demo** set: `arancini`, `serviceradar-agent`, `serviceradar-agent-gateway`, `serviceradar-core-elx`, `serviceradar-datasvc`, `serviceradar-db-event-writer`, `serviceradar-faker`, `serviceradar-flow-collector`, `serviceradar-log-collector`, `serviceradar-rperf-client`, `serviceradar-tools`, `serviceradar-trapd`, `serviceradar-trivy-sidecar`, `serviceradar-web-ng`, `serviceradar-zen`. Typical **farm01** set: `serviceradar-agent`, `serviceradar-agent-gateway`, `serviceradar-core-elx`, `serviceradar-datasvc`, `serviceradar-flow-collector`, `serviceradar-log-collector`, `serviceradar-rperf-client`, `serviceradar-tools`, `serviceradar-trapd`, `serviceradar-trivy-sidecar`, `serviceradar-web-ng`. farm01 usually has no faker, zen, bmp/`arancini`, or k8s-inventory. `serviceradar-log-collector-tcp` shares the log-collector image. Do not invent a separate tag for it. ## Build and push changed images `crane` is on `PATH` (Homebrew). Do not assume `/tmp/gobin/crane`. Staging no longer defines the legacy `remote` / `remote_push` profiles; the surviving ones are `remote_base`, `cache_only`, and `ci`. On a Darwin workstation, running an `oci_push` target through Bazel builds linux/amd64 on RBE and then tries to execute **linux** `jq`/`crane` from runfiles (`Exec format error`). On macOS use `make push_all`, which routes through `scripts/push_all_images.sh` and pushes with host `crane`. Build the OCI layouts remotely, then push with host `crane`: ```bash bazel build \ --config=remote_base \ --noenable_platform_specific_config \ --remote_download_outputs=all \ --compilation_mode=opt \ --stamp \ //docker/images:agent_image_amd64 \ //docker/images:agent_gateway_image_amd64 \ //docker/images:core_elx_image_amd64 \ //docker/images:web_ng_image_amd64 ``` Layouts land under `bazel-out/rbe_platform-opt/bin/docker/images/<name>_image_amd64`. Bazel blob files are often **symlinks** and **mode 0444**; `crane push` of the layout fails with `layout blob ... is a symlink`. Materialize, push by digest, tag **only** `sha-<commit>` (do not retag `latest`): ```bash BIN="$(bazel info output_path)/rbe_platform-opt/bin/docker/images" # or the execroot path printed by the build SRC="$BIN/agent_image_amd64" DEST=/tmp/sr-oci-agent rm -rf "$DEST" mkdir -p "$DEST" rsync -aL "$SRC/" "$DEST/" chmod -R u+w "$DEST" DIGEST=$(jq -r '.manifests[0].digest' "$DEST/index.json") REPO=registry.carverauto.dev/serviceradar/serviceradar-agent crane push "$DEST" "$REPO@$DIGEST" crane tag "$REPO@$DIGEST" "sha-<commit>" rm -rf "$DEST" ``` Repeat for each rebuilt image. Capture every digest. On a Linux host the push targets can be run directly, since the runfiles `jq`/`crane` are the right architecture there. Use `remote_base` for the build; the `remote_push` profile referenced by older runbooks no longer exists. ## Copy unchanged images forward ```bash crane copy \ registry.carverauto.dev/serviceradar/<image>:<old-tag> \ registry.carverauto.dev/serviceradar/<image>:sha-<new> ``` `<old-tag>` is whatever the cluster is running (`v1.4.31`, `sha-<old>`, etc.). If `crane digest` of old and new match, the copy is a retag and existing signatures (if any) still apply. --- ## farm01 roll (no signing) ```bash export KUBECONFIG="$HOME/.kube/farm01.yaml" helm get values serviceradar -n serviceradar helm upgrade serviceradar "$CHECKOUT/helm/serviceradar" \ -n serviceradar \ --reuse-values \ --set image.digests.<service>=sha256:<digest> \ --rollback-on-failure \ --timeout 15m ``` Use `global.imageTag=sha-<new>` ONLY when every first-party image should move. For a change touching one or two services, pin those with `image.digests.<service>` (see "Move only the services you changed") -- farm01 already carries digest pins for `core` and `webNg`, so you are updating an existing pin, not introducing a new mechanism. `--reuse-values` keeps farm01-only settings (MetalLB VIPs, Gateway API attach, Trivy sidecar, empty `registryPullSecret`, `local-path` storage). Do not replace the live values file unless the user wants those template/value changes applied. Use the local chart when `helm/` matches the live chart version (`helm get metadata serviceradar -n serviceradar`). If `helm/` diverged and the user only asked for new images, keep the live chart and only change `global.imageTag` (OCI `oci://registry.carverauto.dev/serviceradar/charts/serviceradar --version <live>` plus `--reuse-values --set global.imageTag=...`). Update `~/src/gitops/clusters/farm01/serviceradar/values.yaml` `global.imageTag` so GitOps matches live. Do **not** commit or push the gitops repo unless the user asks. Verify: ```bash for d in serviceradar-web-ng serviceradar-core serviceradar-agent serviceradar-agent-gateway; do kubectl rollout status deploy/"$d" -n serviceradar --timeout=180s done kubectl get deploy -n serviceradar \ -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.template.spec.containers[*]}{.image}{" "}{end}{"\n"}{end}' kubectl get pods -n serviceradar ``` Done when key deployments show `sha-<new>` and pods are Ready. Report the tag, rebuilt vs copied images, digests, helm revision, and any non-Ready pods. Do not mention signing. --- ## carverauto `demo` roll (sign, then Argo) `demo` admission is Kyverno-enforced. Sign every **rebuilt** digest with OpenBao `cosign-release` **before** changing the Argo app. Copied-forward images whose digest is unchanged keep their existing signatures. OpenBao on the control plane is **HTTPS**. A plaintext `http://127.0.0.1:18200` health check returns `Client sent an HTTP request to an HTTPS server`. ```bash kubectl port-forward -n openbao-system svc/openbao-active 18200:8200 ``` ```bash OPENBAO_ADDR=https://127.0.0.1:18200 OPENBAO_K8S_ROLE=forgejo-signing-runner sa_jwt="$(kubectl create token -n forgejo-actions forgejo-signing-runner)" vault_token="$(curl -skS \ -H 'Content-Type: application/json' \ -d "{\"role\":\"${OPENBAO_K8S_ROLE}\",\"jwt\":\"${sa_jwt}\"}" \ "${OPENBAO_ADDR}/v1/auth/kubernetes/login" | jq -er '.auth.client_token')" export VAULT_ADDR="$OPENBAO_ADDR" export VAULT_TOKEN="$vault_token" export VAULT_SKIP_VERIFY=true export COSIGN_KEY_REF=hashivault://cosign-release export COSIGN_YES=true export COSIGN_DOCKER_MEDIA_TYPES=1 export COSIGN_REFERRERS_MODE=legacy export COSIGN_TLOG_UPLOAD=true ``` ```bash cosign sign --key "$COSIGN_KEY_REF" \ registry.carverauto.dev/serviceradar/<image>@sha256:<digest> ``` Re-mint the Vault token on `403 permission denied`. Tear down the port-forward when signing is done. ### The Argo Application spec is NOT the lever `serviceradar-demo-prod` tracks `helm/serviceradar` on the **`demo/prod-release` branch of the serviceradar repo itself**, and that path contains `.argocd-source-serviceradar-demo-prod.yaml`, whose `helm.parameters` OVERRIDE the Application's own `spec.source.helm.parameters`. Verified 2026-08-23: the Application spec said `image.digests.webNg=sha256:457a2b5c...` while the running pod was `sha256:94bf165f...`, and Argo still reported `Synced`. A `kubectl patch application ... spec.source.helm.parameters` is reverted on the next sync. **Edit the file on `demo/prod-release` and push**, then sync: ```bash git worktree add --no-track -b <tmp> /tmp/wt-demo-release github/demo/prod-release # edit helm/serviceradar/.argocd-source-serviceradar-demo-prod.yaml git push github HEAD:refs/heads/demo/prod-release kubectl patch application -n argocd serviceradar-demo-prod --type merge \ -p '{"operation":{"sync":{"revision":"demo/prod-release"}}}' ``` If the Application parameter and the running image disagree while Argo says `Synced`, that file is why -- do not "fix" it by patching the Application. ### Build from a fresh git worktree Work in a worktree so concurrent agents do not share a checkout. Two things bite on a *fresh* one, both verified 2026-08-23: 1. Symlink the gitignored Bazel rc files before any bazel command (repo Hard Rules). Without them RBE fails with `PERMISSION_DENIED: Missing API key`. 2. **Build once before `make push_all`.** `scripts/push_all_images.sh` resolves `bazel info bazel-bin` and then checks `! -d` on the result *before* it builds anything. On a fresh worktree that directory does not exist yet, so the script dies with: ``` error: unable to resolve bazel-bin ``` which reads like a credentials or config problem and is not. `bazel info` alone succeeds, which makes it more confusing. Prime the output tree first: ```bash bazel build -c opt --config=remote --remote_download_outputs=toplevel //docker/images:images make push_all PUSH_TAG="sha-$(git rev-parse HEAD)" ``` ### Move only the services you changed Use `image.digests.<service>`, not `global.imageTag`, unless every first-party image really should move. `global.imageTag` rolls the whole set for a two-service change; `image.digests.<service>` short-circuits ahead of it in `serviceradar.imageRefSuffix`, so everything else stays on the release tag and only the changed services need signing. Service keys are the `image.tags` names