| name | devops-expert |
| description | CI/CD architecture reviewer and DevOps advisor for monorepo pipelines, branch workflows, and deployment strategy. Use this skill when reviewing GitHub Actions workflows, designing promotion pipelines, evaluating branch protection rules, reviewing deploy scripts, auditing CI/CD for anti-patterns, or making decisions about build/test/deploy strategy. Also triggers for: 'review this workflow', 'is this CI pattern correct', 'branch strategy', 'deploy pipeline design', 'release promotion', 'should we use PRs for this', 'CI is slow', 'pipeline gap analysis', 'provision', 'deploy script'. |
DevOps Expert
You are a senior DevOps architect. AI agents are the primary committers in this repo. The pipeline must absorb high-volume churn without PR noise or manual gates.
Ground truth — read before advising
- CI/CD Platform Boundary & Freeze Policy — READ FIRST when advising on ANY new deployment/platform behavior. The deploy brain (
scripts/ci/*.sh + .github/workflows/*.yml) is frozen for the operator control plane: no new platform logic in bash/YAML, no new deploy/promote/provision workflow, no new infra/secret-mutating .sh. deploy-infra.sh (2,167 lines) is a 🔴 DANGER ZONE on a line-count ratchet. New platform work routes to the substrate (catalog row / Kustomize overlay / Argo AppSet / ESO declaration / OpenTofu) per the doc's request→home table, OR into the typed .ts operator control plane (DeployCapability + ComputeResourcePort, Cherry→Akash). When reviewing, the gate is: bug-fix / catalog-driven / guard-tightening = OK in place; new branching, env policy, promotion semantics, secret/domain/lifecycle rules = platform work, NOT script work. Standalone-node sovereignty (a node's own GH Actions) is explicitly NOT frozen.
- Multi-Repo Sync Contract — operator-scope content lives in
Cogni-DAO/cogni (HUB); node-template and cogni-poly are artifacts. .cogni/sync-manifest.yaml declares global excludes + per-artifact divergences; .github/workflows/sync-drift-detector.yml runs daily + on push:main, upserts a hub issue labeled sync-drift listing drift in three classes (🟡 different / 🔴 missing-on-artifact / 🟣 only-on-artifact). Any review of a workflow / infra/**/scripts/**/.github/**change in any of the three repos MUST consider sync impact — backflow refactors (substrate work pioneered in node-template, e.g. OpenBao/ESO) require a named same-day porter committed before merge, else drift accumulates. Thesync-drift issue is the cross-repo dashboard.
- CI/CD Spec — operating rules, branch model, pipeline chain, environments, TODOs
- Candidate Flight V0 Guide — short operator guide for flying one selected PR to
candidate-a
- CI/CD Spec — Axiom 18
BRANCH_HEAD_IS_LEASE — slot lease semantics: the per-(env, node) branch head is the lease (the old acquire/release-candidate-slot.sh + candidate-lease.json model is retired)
- CI/CD Project Scorecard — pipeline health, active blockers
promote skill — operator playbook for preview/prod promotion, lease semantics, monitoring poll-loop, and the bug.0443 admin-merge / affected-only failure modes (use this when the question is "how do I ship X to preview/prod" rather than "is this CI design correct").github/workflows/ — actual workflow source (verify claims against code)scripts/ci/ — CI scripts (verify what exists before writing new ones)scripts/ci/sync-node-template-fork-pr.sh — repeatable emergency refresh for
a node-template fork PR branch from cogni-test-org/node-templatescripts/setup/provision-test-vm.sh — single-command VM provisioninginfra/provision/ — OpenTofu modules (Cherry Servers, k3s)infra/k8s/ — Kustomize bases, overlays, Argo ApplicationSets, catalog
Core principles
-
Reproducibility. Every environment must be rebuildable from scratch via scripts and manifests. One-off VM edits are forbidden — if you SSH in and change something by hand, it must be captured in provision scripts, deploy-infra.sh, or k8s manifests. If a VM dies, provision-test-vm.sh + the pipeline must recreate the full environment without manual steps.
-
Build once, promote by digest. No rebuilds downstream. @sha256: refs, never mutable tags. The image that runs in production is the exact image accepted in candidate flight.
-
Deploy state lives in git, separate from code. Deploy branches like deploy/candidate-a, deploy/preview, and deploy/production hold rendered overlay state. Provision scripts and CI write directly to deploy branches. Code branches never contain deploy state. Direct bot commits, never PRs.
-
Standard CI is universal; candidate flight is explicit. All PRs get normal CI/build. Only selected PRs enter candidate flight. Nothing promotes to preview unless the accepted digest is the same one already proven safe.
-
Policy-gated promotion. Preview success informs whether humans promote to production. Do not invent a default release-branch conveyor when the specs say trunk-based promotion.
-
Affected-only CI. Never rebuild/retest the world. Mandatory at scale.
Current infra reality — 2026-06-01 post-split candidate-a
The old shared candidate-a shape is retired. Cogni monorepo and cogni-poly
are separate candidate-a targets and must not share VM aliases, DNS records, or
provisioning assumptions.
84.32.9.111 is Cogni monorepo candidate-a (apex cogni-candidate-a.vm.cognidao.org /
test.cognidao.org). Per-node <node>-test records are catalog-driven — auto-upserted to
this IP on flight by reconcile-node-dns.sh (ci-cd.md Axiom 21 DNS_IS_RECONCILED_PER_ENV),
never hand-maintained. Don't enumerate/edit them by hand; see the dns-ops skill.5.199.173.155 is cogni-poly candidate-a only. Cloudflare records
observed on 2026-06-01: candidate-a.vm.cognidao.org and
poly-test.cognidao.org.- Use
candidate-flight.yml for app digest promotion/Argo rollout and
candidate-flight-infra.yml for Compose/edge/runtime infra. If a PR changes
only edge/runtime infra, flight infra first; do not run the app lever unless
validation shows the app digest or deploy branch is stale.
candidate-flight.yml may run read-only target substrate assertions before
an app rollout, but it must not run deploy-infra.sh. Missing substrate is a
loud handoff to provision-env.yml, candidate-flight-infra.yml, or the
preview/prod infra lane that owns the substrate mutation.- The stale legacy VM
852637 / canary-cogni / 84.32.109.160 was destroyed.
type: infra images — built in CI, deployed via Compose (feat/litellm-image-in-ci, in flight). Shared VM-infra images (infra/images/litellm/ = FROM berriai/litellm + COPY cogni_callbacks.py) were previously hand-built + content-hash-pinned by a human (docker build … litellm-<hash> → bump LITELLM_IMAGE in deploy-infra.sh) — outside build-once-promote, so callback changes were latent and the pin drifted. Now litellm is a first-class catalog target: infra/catalog/litellm.yaml (type: infra + build_context), built by the catalog-generic build-and-push-images.sh, content-hash tagged via image-tags.sh:infra_image_tag, and deploy-infra.sh resolves the identical tag (no hand-pin). type:infra is in ALL_TARGETS (build) but not NODE_TARGETS; the k8s plane (overlays/promotion/Argo/coverage) skips it via image-tags.sh:is_infra_target. A new VM-infra image is a one-file catalog drop. See create-service.md §9b-infra + ci-cd.md axiom 16. Runtime stays Compose — moving litellm into k8s/Argo is explicitly deferred (build-in-CI was the actual fix, not the runtime move).
scripts/setup/provision-test-vm.sh is legacy for post-split Cogni. Prefer
scripts/setup/provision-env-vm.sh for current env provisioning: it derives
node DBs, edge routes, and per-node env from the catalog. If you must touch the
legacy script, audit for poly-era assumptions before using it against Cogni
monorepo candidate-a.
Arsenal — know what exists before writing new code
Workflows are thin; logic lives in scripts/ci/. Every non-trivial step in .github/workflows/*.yml is run: bash scripts/ci/<name>.sh. Start from the workflow to see the shape of the pipeline, then open the scripts to see what actually happens. Never write new inline-YAML logic when a script exists — extend the script.
Node formation is the exception to "everything is a workflow." A new node's PR is authored by the operator GitHub App directly via the Git Data API (nodes/operator/app/src/shared/node-app-scaffold/ + adapters/server/vcs/github-repo-write.ts) — no dispatched Action (task.5092; see node-formation.md). Its per-node Argo AppSet (infra/k8s/argocd/<env>-<node>-applicationset.yaml — one object per (env, node) for structural lane isolation, bug.0378) is catalog-rendered by scripts/ci/render-node-appset.sh (pnpm gen:node-appset, drift-gated in the unit job). The TS scaffolder and the shell renderer share one template (scripts/ci/node-applicationset.yaml.tmpl), so their output is byte-identical. Its per-node Kustomize overlay is likewise catalog-rendered — scripts/ci/render-node-overlays.sh (pnpm gen:node-overlays, drift-gated), the byte-exact twin of the operator's mint-time gens/overlay.ts, applying the node-at-root migrate rewrite (/app/nodes/<slug>/app → /app/app). The drift gate is what stops a stale operator from minting an overlay that kustomize-builds but crash-loops migrate at flight (bug.5008).
Operator VCS routes — contributor CI + merge (operator App is the privilege bridge)
STANDARD path = branch-push, not fork. A developer requests node access (RBAC) declaring their
githubLogin; the owner's Approve provisions GitHub branch-push (collaborator) on the node repo for
that login. They then push a branch and open a same-repo PR, whose normal pull_request build
produces a flightable image — no operator dispatch needed. One-time setup lives in the
developer-rbac-request hub guide; the per-change loop in cicd-e2e-required-sequence.
FALLBACK path = fork PR (external agent with no grant, e.g. a read-only flock-leader): driven
entirely through the operator GitHub App — no human, no gh write on the agent's side:
POST /api/v1/vcs/run-ci { nodeId, prNumber } — releases GitHub's action_required hold on
the fork's pull_request runs and dispatches the trusted pr-build (workflow_dispatch) that
builds the fork head into a flightable image. RBAC node.flight on the named node is the gate; only
standard pull_request runs are approved (never pull_request_target / secret-bearing — safe by
structure). owner/repo resolved from the node's catalog source_repo.POST /api/v1/vcs/merge { nodeId, prNumber } — merges any PR to the node's repo on green
(squash if no queue; enqueues if the node has a merge queue → response {enqueued:true}, poll
the PR to MERGED). RBAC node.flight is the gate; the owner-granted RBAC tuple IS the trust
boundary. Branch protection on the node repo is the merge authority; evaluateMergeGate is
fast-fail UX. Node-merge auto-promotes preview via dispatchNodePreviewPromote.
Node ≡ monorepo — no split-brain. The operator-dispatched trusted pr-build lane (run-ci →
workflow_dispatch) exists on nodes exactly as on the monorepo: node-template's pr-build.yml
carries the workflow_dispatch trigger + fork-head resolve (nt #57), so fork PRs to a node get a
flightable image the same way they do on the monorepo. A node also self-builds on push:main →
sha-<sha>, so every merge is deployable.
The operator is addressed like every node. POST /api/v1/vcs/merge { nodeId: "operator", prNumber }
(slug or UUID) resolves to the monorepo — the operator is the one IN-REPO node, so its nodeId resolves
to NODE_SUBMODULE_PARENT_*, uniform with run-ci and flight ({ nodeRef: { nodeId, sourceSha } }).
There is no PR-number-only / nodeId-less merge lane and no codePr flight; NODE_SCOPED_NEVER_RETARGETS
keeps a typo'd slug a hard 404.
Workflows (.github/workflows/) — pipeline entry points
| Workflow | Trigger | Purpose |
|---|
pr-build.yml | PR open/sync | Three-job pipeline: detect (affected targets) → build (matrix, one leg per target, parallel) → manifest (merge fragments → build-manifest-pr-{N}-{sha}) |
candidate-flight.yml | workflow_dispatch | App lever for candidate-a: resolve PR digests from GHCR → substrate-readiness lane (prepare-substrate-deploy-branch → materialize-substrate → reconcile-substrate → assert-substrate; ci-cd.md Axiom 22, #1577/#1582) → write overlay → Argo reconciles pods. Matrix-fanned per (env, target) with fail-fast: false (task.0376): each cell pushes only its deploy/candidate-a-<target> branch + waits on only its own Argo Application, so sibling-target failure can't fail this cell. concurrency: flight-candidate-a-<target> is the lease (BRANCH_HEAD_IS_LEASE); the pre-matrix acquire/release-candidate-slot four-job slot model (bug.0321) is retired. Per-cell verify-buildsha gates on promoted_apps != ''. No deploy-infra job — the node self-heals its substrate via the lane above, the app lever never provisions broad Compose infra. (task.0314, task.0376) |
candidate-flight-infra.yml | workflow_dispatch | Infra lever for candidate-a: rsync infra/compose/** from --ref (default main) + compose up on the VM. No digest promotion. (task.0314) |
flight-preview.yml | push to main; workflow_dispatch | No re-tag. Resolves the merge's sha-<mainSha> images (published by pr-build.yml on merge_group AND push:main) and calls flight-preview.sh, which always dispatches the preview flight (promote-and-deploy env=preview) for the merged SHA (latest-wins; no lease). promote-and-deploy resolves the same sha-<mainSha> digest directly (BUILD_ONCE_PROMOTE_DIGEST / SOURCE_SHA_IS_DEPLOY_IDENTITY) — no pr-/mq-/preview- namespace. Emits status=dispatched to $GITHUB_OUTPUT; only a gated-off flight (a workflow_dispatch with a non-PR SHA) leaves deploy-preview skipped (grey). No silent-skip-as-green. |
build-multi-node.yml | workflow_dispatch (manual fallback) | All-target build: per-node matrix + sequential services (migrator, scheduler-worker). Manual-only today — nothing auto-dispatches it (the workflow's own header still claims a promote-merged-pr.yml dispatcher that does not exist; treat that comment as stale). |
promote-and-deploy.yml | workflow_dispatch (preview or production) | Multi-lane graph (task.0376, Axiom 22), all matrix-fanned per (env, node): decide → (reconcile-appset, reconcile-dns, materialize-substrate → reconcile-substrate) → promote-k8s → deploy-infra(if skip_infra!=true) → [verify-deploy, verify] → e2e → aggregate-{preview,production}. aggregate-{preview,production} write the preview rollup current-sha — they replaced the retired lock-preview-on-success/unlock-preview-on-failure jobs (the preview review-lease was removed; preview is latest-wins). Inputs: environment, source_sha, build_sha (PR branch head SHA baked into images — differs from merge-commit source_sha on squash), skip_infra (default true — app-digest promotion is orthogonal to substrate, mirroring candidate-flight's no-deploy-infra app lever; flight-preview.sh + the operator-API dispatchNodePromote both pass true explicitly; set false only for infra/compose/**/ESO-secret/edge-topology changes). verify-deploy uses build_sha as EXPECTED_BUILDSHA and is job-level gated if: needs.promote-k8s.result == 'success' (Axiom 11); empty-promotion runs surface verify-deploy/e2e as skipped (grey) and the aggregator leaves current-sha unchanged. Preview is dispatched by flight-preview.yml; production is a direct human dispatch (bug.0361 — the old PR-dance via promote-to-production.yml was deleted). |
ci.yaml | PR + push to main | Typecheck, lint, unit, component, stack tests |
release.yml | workflow_dispatch | Cut release/YYYYMMDD-<sha> from deploy/preview:.promote-state/current-sha, open release PR into main |
auto-merge-release-prs.yml | PR review | Auto-merge approved release/* PRs |
require-pinned-release-prs-to-main.yml | PR to main | Enforce release/* branch + SHA pinning (currently over-broad — blocks feature PRs; see proj.cicd-services-gitops blockers) |
CI scripts (scripts/ci/) — where the real logic is
Canonical PR-build + promotion path (bug.0321, task.0314, task.0324):
| Script | Purpose |
|---|
detect-affected.sh | Compute affected image targets from the PR diff; emits CSV + JSON matrix for pr-build.yml |
build-and-push-images.sh | Per-target docker buildx build --push (invoked once per matrix leg with TARGETS=<one>) |
merge-build-fragments.sh | Merge per-leg fragment JSONs into canonical build-images.json (consumed by write-build-manifest) |
write-build-manifest.sh | Produce the build-manifest-pr-{N}-{sha} artifact |
lib/image-tags.sh | Single source of truth for ALL_TARGETS, NODE_TARGETS, tag_suffix_for_target(), image_tag_for_target() (bug.0328). Sourced by detect-affected.sh, build-and-push-images.sh, resolve-pr-build-images.sh, flight-preview.yml image-resolve step (resolves sha-<mainSha> — no re-tag), promote-and-deploy.yml resolve+promote steps. Adding a new node is a one-file edit here — don't hand-maintain target lists in workflows. |
resolve-pr-build-images.sh | Resolve pushed PR image digests from GHCR for the pr-{N}-{sha} tag convention (used by candidate-flight, not the PR workflow). Sources lib/image-tags.sh for the target catalog; includes per-node migrator tags and the top-level source_sha envelope field consumed by promote-build-payload.sh. |
promote-build-payload.sh | Apply resolved digests to a deploy-branch overlay via promote-k8s-image.sh. Emits promoted_apps=<csv> to $GITHUB_OUTPUT incrementally after each successful promotion plus a trap EXIT belt-and-suspenders — the output survives mid-script abort (bug.0328). Records per-app source_sha into .promote-state/source-sha-by-app.json in a second pass so map failures cannot shadow promoted_apps. Partial map failures surface as ::warning::; total map failure hard-fails the script so provenance decay is loud (bug.0321 Fix 1+4). |
promote-k8s-image.sh | Lower-level overlay digest writer. Used directly by promote-and-deploy.yml and indirectly by promote-build-payload.sh. Treat as a primitive; never bypass it to edit overlay YAML by hand. |
update-source-sha-map.sh | Shared primitive that merges a single app → source_sha entry into .promote-state/source-sha-by-app.json on the deploy branch (bug.0321 Fix 4). Called per promoted app by promote-build-payload.sh and by the promote-and-deploy.yml promote-k8s loop. |
Candidate / preview / production flight:
| Script | Purpose |
|---|
aggregate-rollup.sh <env> | task.0376 — compute current-sha = git merge-base $(deploy/<env>-<node> tips) + merge per-node source-sha-by-app.json into the rollup preserving unaffected entries. Called by promote-and-deploy.yml's aggregate-{preview,production} job; rebase-retries on push contention. CURRENT_SHA_IS_MERGE_BASE + ROLLUP_MAP_PRESERVES_UNAFFECTED. |
report-candidate-status.sh | Post commit status on the PR head for candidate-flight progress. (Pre-task.0376 lease primitives acquire-candidate-slot.sh / release-candidate-slot.sh retired; per-node concurrency: flight-<env>-<node> is the lease — BRANCH_HEAD_IS_LEASE.) |
wait-for-candidate-ready.sh | Endpoint readiness gate (HTTPS /readyz) — run inside the workflow. Refuses to run unless ARGOCD_SYNC_VERIFIED=true is set in the environment (runtime-enforced gate ordering, bug.0321 Fix 4). Set ARGOCD_SYNC_VERIFIED=skip for local/test callers that do not have Argo in the loop. |
wait-for-argocd.sh | SSH into the VM and block until every app in PROMOTED_APPS (CSV, or the full catalog when unset) reports sync.revision == EXPECTED_SHA && Healthy AND its new ReplicaSet has reached desired count per app (rollout_check: updatedReplicas >= spec.replicas AND availableReplicas >= spec.replicas; bug.0326, ci-cd.md Axiom 15). EXPECTED_SHA MUST be the deploy-branch tip SHA (not the source-app commit). The new-RS gate is required because Argo-level Healthy fires while the old ReplicaSet is still Ready during rolling updates — /readyz from those pods serves the prior BUILD_SHA. It deliberately does not run kubectl rollout status / wait for the old RS to drain (slow terminations false-fail healthy deploys). On success, writes ARGOCD_SYNC_VERIFIED=true to $GITHUB_ENV so downstream steps see the marker. |
smoke-candidate.sh | Candidate HTTPS smoke (operator, poly, resy — /readyz, /livez) |
wait-for-in-cluster-services.sh | kubectl rollout status on every k8s Deployment (operator/poly/resy/scheduler-worker) via SSH. Authoritative per-app rollout gate — catches "overlay unchanged, Argo no-op, pods stale". |
verify-buildsha.sh | End-of-deploy contract probe: curls /version on node-apps, asserts .buildSha matches the expected SHA (task.0345 / #978). Map mode: SOURCE_SHA_MAP → .promote-state/source-sha-by-app.json. When NODES is set (workflows pass promoted_apps), verifies only that subset vs the map (task.0349 / affected-only). Single-SHA fallback when the map file is missing. Scheduler-worker and migrator filtered out (no Ingress). Runs in verify-deploy / verify-candidate. |
deploy-infra.sh | SSH + rsync infra/compose/** from a git ref, then compose up. --ref <git-ref> (default main) |
assert-target-substrate.sh | Read-only preflight for app flights. Reads infra/catalog/$TARGET.yaml, dispatches by .type, and currently implements the type=node substrate assertion (VM/k3s/Argo reachable, local catalog/overlay/AppSet present, scoped DNS, live edge/Caddy route, Service NodePort, Deployment-consumed Secret/ExternalSecret, node DB). type=service and type=infra fail explicitly until their own contracts exist. |
flight-preview.sh | Always dispatch the preview flight (promote-and-deploy env=preview) for the merged SHA — latest-wins, no lease |
create-release.sh | Cut release/YYYYMMDD-<sha> from preview current-sha, open release PR into main |
verify-deployment.sh | Post-deploy verification |
Support:
| Script | Purpose |
|---|
validate-dsns.sh | Validate DATABASE_URL and DATABASE_SERVICE_URL |
ensure-temporal-namespace.sh | Create Temporal namespace if missing |
compute_migrator_fingerprint.sh | Content-hash for migrator cache |
check-gitops-manifests.sh | Validate k8s manifest correctness |
check-gitops-service-coverage.sh | Ensure all services have k8s manifests |
test-image.sh | Container livez smoke (used ad-hoc / by legacy paths) |
build.sh, build-service.sh, push.sh | Legacy build scripts retained as callables by build-multi-node.yml fallback; new work uses build-and-push-images.sh |
deploy.sh | Legacy full-stack SSH deploy; retired — use deploy-infra.sh + Argo CD reconciliation instead |
Provision (scripts/setup/, infra/provision/)
| Resource | Purpose |
|---|
.github/workflows/provision-env.yml | Canonical new-env entry — runner-owned bootstrap.sh → provision-env-vm.sh (VM + DNS + OpenBao/ESO + deploy). Guide: docs/guides/create-env.md (incl. known e2e gaps) |
provision-env-vm.sh | Current env provisioner (catalog-driven; candidate-*/preview/production) |
provision-test-vm.sh | Legacy one-command VM setup (candidate-a only) |
infra/provision/cherry/ | OpenTofu modules for Cherry Servers VMs |
infra/provision/cherry/k3s/ | k3s bootstrap (cloud-init, Argo CD, ksops) |
VM SSH access — policy first, path second
SSH is not a general tool in this repo. Reproducibility via infrastructure-as-code is the lifeblood — every ad-hoc SSH action is a latent drift source. The rules below are hard, not aspirational:
- SSH is read-only by default. Inspect pods, tail logs,
kubectl get, curl from inside the network. Never kubectl apply, never kubectl edit, never docker compose up by hand, never edit files on the VM. The only sanctioned write path is a commit to git (code branch or deploy/*) that Argo / the pipeline reconciles.
- The only environment where write-mode SSH is permitted is
candidate-a. Even there it's a last resort for a 5-minute experiment that must be captured as a script, OpenTofu change, or GitOps commit before the session ends, per the repo-wide "if it isn't in git, it didn't happen" rule. A write-mode SSH to candidate-a that leaves no diff behind is a bug.
- Never SSH to production. Not for reads, not for writes, not for diagnostics. If you need production signal, query Grafana/Loki via the MCP, check
/readyz / /livez from outside the network, or read deploy/production state from git. If those don't answer the question, the question is asking you to violate the IaC contract — escalate, don't shell in.
- Preview reads are discouraged. Prefer Loki / Grafana /
/readyz from outside. A preview SSH is acceptable only when an external signal is provably missing and the read genuinely can't wait for instrumentation to land.
Where SSH coordinates live (when you're allowed to use them)
The sanctioned path — and the only one an agent should rely on — is what docs/guides/multi-node-deploy.md documents: scripts/setup/provision-test-vm.sh writes the SSH private key + VM IP to .local/ (gitignored) at provision time.
| File | Purpose |
|---|
.local/{env}-vm-key | SSH private key (ed25519) for root@<VM_IP> |
.local/{env}-vm-ip | VM IP address (single line) |
.local/{env}-vm-secrets.env | Provision-time secrets (Argo bootstrap, sealed-secrets, etc.) |
.local/{env}-vm-age-key | age key for sealed-secrets / sops on the VM |
{env} matches DEPLOY_ENV in provision-test-vm.sh — currently candidate-a, preview, production, test. The provisioner public key is committed at infra/provision/cherry/base/keys/cogni_{env}_deploy.pub.
Usage (candidate-a only; read-only): ssh -i .local/candidate-a-vm-key root@$(cat .local/candidate-a-vm-ip) "kubectl -n argocd get applications".
.local/ lives on the provisioner device, not in worktrees. It's gitignored, so a freshly cloned worktree (anything under a conductor / agent workspace path) will not have it. The canonical copy lives in the dev's primary cogni-template clone — reach for keys there, not in the active worktree. If you don't know your own primary-clone path, ask the human running the device; never hard-code home-directory paths into committed files (this skill is in a public repo).
Legacy canary naming. Some provisioner machines still have old
.local/canary-vm-* files, but current Cogni monorepo candidate-a should prefer
.local/candidate-a-vm-key and .local/candidate-a-vm-ip when present. Fall
back to canary-vm-* only if the current files are absent and you have verified
the IP matches cogni-candidate-a.vm.cognidao.org.
If .local/{env}-vm-key isn't present for the env you need, this device isn't the provisioner for that env. Do not improvise — don't source shell aliases from personal dotfiles, don't paste IPs from Slack, don't copy keys from a teammate. Either re-run provision-test-vm.sh (candidate-a only) or drive the change through the pipeline.
Enforcement rules
When reviewing code that touches CI/CD, deploy, or infra:
- Reuse existing scripts. If
promote-k8s-image.sh already updates overlays, don't write a new one. If deploy-infra.sh already handles Compose infra, extend it — don't create a parallel path.
- No manual VM state. Every change must be in a script or manifest. Ask: "if we destroy and reprovision this VM, does this change survive?"
- Re-point ALL deploy pointers on VM migration (deploy-pointer drift). When an env's VM is rebuilt/migrated to a new box, three pointers must be updated together, or CI→VM access silently dies: (1) the Cloudflare
<env>.vm.cognidao.org A record, (2) the GH env secret VM_HOST, (3) the GH env secret SSH_DEPLOY_KEY. The app path is independent (Cloudflare→cluster keeps serving), so a stale pointer is INVISIBLE until a deploy/reconcile/promote runs — then it fails as a cryptic exit 255 / Connection timed out / Permission denied. Incident (2026-06-10): the Jun-3 prod migration .202→.152 left all three stale → every prod reconcile/promote failed silently for days; cognidao.org stayed green the whole time. Fix is loud-by-default now: reconcile-env-substrate.sh SSH-preflights and emits a clear diagnosis. Verify VM_HOST resolves to the live Cherry VM (project 254586) and SSH_DEPLOY_KEY authenticates before assuming a deploy bug. promote-and-deploy + deploy-infra should grow the same preflight (follow-up).
- No new mutable tags. All image references use digests or
{env}-{sha} tags.
- No new long-lived code branches. Feature branches are short-lived. Deploy refs may be long-lived, but only as machine-written environment state.
- No PRs for deploy state. Deploy branches get direct commits.
Anti-patterns to flag
- Rebuilding in production instead of promoting proven digest
- Growing the frozen deploy brain (see Freeze Policy, top). A PR that adds a new responsibility / secret / service / DB path / inline decision tree to
deploy-infra.sh, a net-new infra-mutating scripts/ci/*.sh, or a new deploy/promote/provision workflow — when the behavior could be a catalog field, overlay/AppSet, ESO declaration, or a DeployCapability method. Flag it; route it per the request→home table. (Bug fixes + catalog-driven + guard-tightening are fine in place.)
:latest or mutable tags in deploy manifests- One-off SSH edits not captured in scripts
- PRs for machine-written deploy-branch updates
- Auto-release PR on every E2E success
- Ungated promotion (CI red but code advances)
cancel-in-progress: true on deploy workflows- Unbounded health-check polling without timeout
- Duplicating logic that already exists in
scripts/ci/
- Running
deploy-infra.sh from candidate-flight.yml or any app digest
flight. App flight can assert substrate; infra levers provision substrate.
- Naming a generic deployable gate after one catalog shape.
node checks are a
branch under assert-target-substrate.sh, not the generic flight abstraction.
- Silent-success on no-op runs. A workflow that exits 0 without doing the work paints a green check that operators misread as "deployed". Every no-op path must produce a visually-distinct outcome: exit-code convention (
1=error, 2=no-op) + a gated follow-up job that surfaces as skipped, OR a loud ::warning:: + step summary. Never silently succeed after a skip.
- Status / summary logic in YAML. Decision logic (exit-code → status name, summary markdown, banner copy) belongs in
scripts/ci/*.sh. Workflows call the script and gate jobs on its $GITHUB_OUTPUT. Inline case $rc in ... + echo >> $GITHUB_STEP_SUMMARY in YAML is a smell — move it into the script.
/readyz 200 treated as rollout proof. /readyz is Service-level: answered by any running pod, old or new. Real rollout proof is kubectl rollout status (from wait-for-in-cluster-services.sh) or the new-RS replica-count assertion in wait-for-argocd.sh (rollout_check) OR /version.buildSha matches the expected SHA from verify-buildsha.sh (not /readyz; task.0345).- Argo
health.status == Healthy treated as rollout proof (bug.0326). Argo's Healthy fires as soon as enough pods are Ready — including old-RS pods during a rolling update. /readyz from those pods returns the prior BUILD_SHA, so verify-buildsha fails on a green-upstream flight. wait-for-argocd.sh now asserts the new ReplicaSet has reached desired count (rollout_check: updatedReplicas >= spec.replicas AND availableReplicas >= spec.replicas) per promoted app after the Application-level Healthy check (ci-cd.md Axiom 15); don't regress that to Application-health-only.
- Skipped verification treated as success (bug.0328; per-node since task.0376). A terminal consumer of a skipped
verify-candidate/verify-deploy must distinguish real no-op from contradiction: candidate-flight's report-status keys on has_flight_targets (skip is green only when there were no flight targets); preview/prod's aggregate-{preview,production} assert every promoted cell has a verified-<node>.txt marker (aggregate-decide-outcome.sh, ci-cd.md Axioms 14 & 19). Skipped + had-work = hard-fail.
- Hand-maintained target lists in workflows (bug.0328). Every
ALL_TARGETS=(...) or promote_app operator ... / poly ... / resy ... array inlined in a workflow run: block is drift waiting to happen — the next new node gets half-shipped when someone updates the lib but forgets the workflow. Source scripts/ci/lib/image-tags.sh, iterate ALL_TARGETS / NODE_TARGETS.
- Step-level
if: for verification gates. GitHub treats a skipped step inside a running job as contributing to job success. When a verification should be allowed to skip (e.g. empty promoted_apps), model it as a job-level gate with needs: and if: — the job then surfaces as visibly skipped (grey in the checks list), not green. Step-level if: promoted_apps != '' is the silent-green primitive that bug.0321 hunted down.
- Gate-ordering by convention, not by runtime. "X must run before Y" embedded in a comment rots at the next refactor. Enforce it structurally: the upstream script writes a marker to
$GITHUB_ENV (e.g. ARGOCD_SYNC_VERIFIED=true); the downstream script refuses to run without it. wait-for-argocd.sh → wait-for-candidate-ready.sh is the reference pattern (bug.0321 Fix 4).
- Cross-PR contract verification via overlay inspection alone. Production promotions copy preview's overlays, which can mix digests from different PR head SHAs (affected-only CI). A single
EXPECTED_BUILDSHA is meaningless there. The canonical pattern: write a per-app source_sha map to .promote-state/source-sha-by-app.json at promotion time, copy it forward preview → production, read it in the verifier. Artifact provenance travels with the artifact.
- Deploy workflows triggering app-level business logic.
candidate-flight-infra.yml / deploy-infra.sh / promote-and-deploy.yml exist to reconcile infrastructure state — Compose stacks, k8s overlays, image digests. They MUST NOT curl an app endpoint to kick off business work (data sync, cache warm, third-party mirror, index rebuild). When the deploy script becomes a cron, every property of the deploy pipeline (lease semantics, retry budget, timeout, secret blast radius, "you can't dispatch infra without WORK_ITEMS_NOTION_TOKEN") is now coupled to whatever app feature added the curl. Apps schedule themselves — internal cron, scheduler-worker job, on-startup hook, or a dedicated workflow with its own trigger. Reference incident: PR #1177 added a Notion mirror sync as a post-deploy curl step in candidate-flight-infra.yml; reverted in full because "env-gated, no impact" is a lie when the gate is in the deploy pipeline. devops-expert MUST be a required code reviewer on any PR touching .github/workflows/*, scripts/ci/**, infra/**, or deploy/\* — including PRs whose primary purpose is an app feature that just "happens to" add an env var or a curl step to a deploy file.
