A full provision is the default. The narrow, idempotent day-2 heal mode reconciles the OpenBao SA/role substrate on an existing VM without a full re-provision (verify-and-heal; a re-run is a no-op):
-
Apex cutover is in Phase 4b, BEFORE apps deploy (Phase 7). For production, cogni_operator_domain_for_env(production) = cognidao.org (apex) — provisioning repoints the live apex to the new, app-less VM for 10–20 min. A failed run leaves the apex on a dead VM (a DNS time-bomb under resolver cache). Always have the old VM IP and roll the apex back if it goes sideways. Cleaner: blue-green — provision with DOMAIN=<env>-next.<root> override, validate the operator fully, then flip the apex by hand.
-
Placeholder image digests = ImagePullBackOff that NEVER self-heals. Nodes the seed-source doesn't run get ghcr.io/.../cogni-template:<env>-placeholder-<node> (a fake tag). Each node has its OWN image (same repo name, different digest) — do NOT reuse one node's digest for another (node-template's lacks nodes/operator/app/migrate.mjs → MODULE_NOT_FOUND in the migrate init container). Fix: get the correct per-node digest from a healthy env (KUBECONFIG=.local/candidate-a-kubeconfig.yaml kubectl get deploy -n cogni-candidate-a -o custom-columns=NAME:.metadata.name,IMG:.spec.template.spec.containers[0].image), patch each deploy/<env>-<node> overlay (infra/k8s/overlays/<env>/<node>/kustomization.yaml: replace newTag: "*placeholder*" with newName + digest), push, kubectl -n argocd annotate app <env>-<node> argocd.argoproj.io/refresh=hard --overwrite. The real fix = provision seeds real per-node digests (or run /promote).
-
local_certs missing → Caddy 525 on every public host. The rendered Caddyfile has bare site blocks (no TLS directive) → Caddy attempts Let's Encrypt behind the Cloudflare proxy → challenge can't reach origin → no cert → tlsv1 internal error → CF returns 525. The zone SSL mode is full, so the fix is local_certs (self-signed, CF Full accepts it). Manual: add local_certs inside the global { } block of /opt/cogni-template-edge/configs/Caddyfile.tmpl on the VM (before the log { line), docker restart cogni-edge-caddy-1. Durable fix = scripts/ci/render-caddyfile.sh emits local_certs in its global block. (A pre-existing env may "work" only because it got an LE cert during an old grey-cloud window.)
-
H7 — preview/prod deploy-branch divergence. preview/production refuse force-update; Phase 4c commits env-state.yaml on top of the seed, so the tip is always ahead → re-runs abort at Phase 4b.5. Clear deploy/<env>* branches before re-dispatch (candidate-* auto-force, so this only bites preview/prod). They re-seed clean.
-
Re-run trap. "Restore prior init-artifact keys" aborts if a previous run uploaded artifacts under a different passphrase. Deleting the Cherry VM + SSH key is NOT sufficient — the restore step keys off the prior run's GH <env>-init-artifacts (1-day retention), not the VM. So a fresh-passphrase re-dispatch still finds the old artifact and dies decrypt failed for <env>-vm-key. Fixes: (a) keep the SAME passphrase across retries of one env (simplest — never regenerate mid-cycle), or (b) delete the prior artifact: gh api repos/<owner>/<repo>/actions/artifacts --jq '.artifacts[]|select(.name|test("<env>-init-artifacts"))|.id' → gh api -X DELETE …/artifacts/<id> → re-dispatch (fresh-key path). Seen 2026-06-04: regenerating the passphrase between candidate-b retries wasted a dispatch.
-
Quoted secret values brick init. gh secret set of a value that includes literal quotes ('https://...') stores the quotes → URL-validated fields (EVM_RPC_URL, POSTHOG_HOST) fail the Zod schema → /readyz 503 init-loop. .env.* files are quoted (fine for source, not for verbatim gh secret set). Strip quotes before setting; operator-200 on the old prod proves nothing about the new secrets path.
-
.env.bootstrap extraction traps. Lines carry inline comments and a commented stale value, and GITHUB_ADMIN_* (file) vs GH_ADMIN_* (GH secret) differ. Extract with grep -m1 "^KEY=" f | cut -d= -f2- | awk '{print $1}' | sed "s/['\"]//g" (first field, quote-stripped) — naive cut -d= -f2- grabs the trailing comment.
-
You CANNOT debug app failures from gh run. The run shows phase text only; the real failure (ImagePullBackOff, init MODULE_NOT_FOUND, 525) lives in the cluster and Loki. Two access paths:
- kubeconfig from the init artifacts (after the run ends):
gh run download <id> --name <env>-init-artifacts, decrypt each *.enc with openssl enc -d -aes-256-cbc -pbkdf2 -iter 200000 -pass pass:<PP>, then KUBECONFIG=.../<env>-kubeconfig.yaml kubectl ....
- Loki (no cluster access needed, works mid-run): prod ships k8s-events to Grafana Loki —
{env="<env>"} |~ "(?i)imagepull|backoff|oom|evicted|context deadline". This is how you read the real pull error while Phase 9 is still grinding.
-
Argo selfHeal=true reverts kubectl hacks. kubectl set image / edit gets reverted to the deploy-branch state within ~3 min. Fix the deploy branch (push) — that's the source of truth.
-
GHCR throttle ≠ placeholder. A fresh 6gb-shared VM pulling 5 images at once can transiently BackOff (recovers). A placeholder tag BackOffs forever. Distinguish via Loki: a :<env>-placeholder-<node> tag is the never-heals case; a real @sha256:... that's slow is transient.
-
/readyz cascade. Every node's /readyz asserts scheduler-worker connectivity. If scheduler-worker is down (placeholder/ImagePull), all nodes report 0/1 + Phase 9 fails everything — looks like 5 failures, is 1 root cause. Fix scheduler-worker → the rest go Ready.
-
Postgres root-password drift wedges the WHOLE infra deploy (and looks like an alloy/observability failure — it isn't). POSTGRES_PASSWORD is honored only at first-init of an empty volume. After a re-provision or a POSTGRES_ROOT_PASSWORD rotation, the persisted postgres_data volume keeps the old password while deploy-infra renders .env from the GH-secret SSOT → db-provision's TCP+scram auth fails, spins its full 120s, and deploy-infra.sh aborts at the db-provision step. The ERR-trap diagnostic then dumps alloy "unsupported protocol scheme" + "unhealthy containers" — red herrings: alloy/s-ui/git-sync/repo-init/alloy-k8s-events have no healthcheck (Docker reports null, not unhealthy), and the empty-Loki-URL line is a transient of that one re-render. The real line is ❌ Timed out waiting for Postgres + FATAL: password authentication failed for user "postgres". Downstream blast radius: the abort happens before deploy-infra's Step-7 k8s-Secret write, so the <node>-node-app-secrets Secret is never refreshed → every catalogued external key (GH_REVIEW_APP_*, DOLTHUB_*, PRIVY_*, LANGFUSE_*, OAuth) stays empty in the pod even though it's present in the GH env + carried in the promote-and-deploy env block. Fix is in git (idempotent, self-healing): deploy-infra.sh reconciles the superuser password via the postgres image's local … trust unix socket (compose exec … ALTER USER) before db-provision, and postgres-init/provision.sh re-asserts the app/service role passwords on the "already exists" branch. Manual unwedge if you hit an unpatched env: docker exec cogni-runtime-postgres-1 psql -U postgres -c "ALTER USER postgres PASSWORD '<.env value>'" (and app_user / service user), then re-run the deploy.
-
Phase 5f db-backup (exit 137) aborts the ENTIRE deploy-infra → zero app layer. deploy-infra.sh:643 runs $RUNTIME_COMPOSE --profile backup up --force-recreate --no-deps --abort-on-container-exit --exit-code-from db-backup db-backup. On a fresh/loaded VM the db-backup container is SIGKILLed (137) and --abort-on-container-exit kills deploy-infra before Step-7 → the cogni-<env> namespace, the <node>-node-app-secrets, and the per-node Argo Applications are never created. Symptom: argocd/openbao/external-secrets/reloader all Synced+Healthy, zero app pods, no cogni-<env> ns, URLs 502/000. Looks like total node failure; is one backup container. (Distinct from Gotcha 12, which aborts at db-provision.) Seen 2026-06-04 — db-backup needs a timeout/retry or a skip-on-empty-DB guard so a flaky backup can't gate the whole app layer.
-
A provision that dies late can upload an init-artifact INCONSISTENT with the live cluster — root + SSH lost despite a "good" artifact. Seen 2026-06-04: <env>-init-artifacts decrypted cleanly and its kubeconfig authenticated to the live k3s, but the same artifact's openbao-init.json root_token did not authenticate (bao token lookup -self fails on an unsealed OpenBao), its vm-key did not SSH (publickey denied), AND generate-root with its unseal_keys_b64 failed. A clean decrypt is NOT proof of valid custody. Validate immediately: kubectl -n openbao exec openbao-0 -- env BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=$RT bao token lookup -self. If it mismatches, the env has no recoverable root or SSH — surgical recovery is impossible; destroy + reprovision is the only path. (Likely cause: auto-init/auto-unseal racing the script's bao operator init, or a VM/key regen between phases — capture happens at a different moment than the live state.)
-
Reprovision reuses the Cherry IP but rotates BOTH the host key and the deploy key. Fresh VM at the same IP → (a) local known_hosts MITM warning, clear with ssh-keygen -R <ip>; (b) the GH-secret VM deploy key goes stale → candidate-flight-infra (every infra-lever workflow) fails at "Setup SSH" (publickey) until that secret is refreshed with the new <env>-vm-key. So you can't re-run deploy-infra via the workflow against a freshly-reprovisioned VM until you update the deploy-key secret — and if Gotcha 14 hit, you have no working key to update it with.
-
Phase 1 of the DB-cred OpenBao port (<env>-db-reader) validates via k8s-auth — NO root token needed. Even on a provision that died at 5f, if Phase 5b completed, the <env>-db-reader policy/role + db-provisioner SA are live. Prove the OpenBao read path with the kubeconfig alone: JWT=$(kubectl create token db-provisioner -n default); TOK=$(kubectl -n openbao exec openbao-0 -- env BAO_ADDR=http://127.0.0.1:8200 bao write -field=token auth/kubernetes/login role=<env>-db-reader jwt=$JWT) → bao kv list cogni/<env> succeeds (least-priv read). This is how Invariant-15 Phase 1 was deploy-verified on candidate-a despite the dead app layer.
-
$REPO_ROOT (or any runner-only var) referenced INSIDE the remote heredoc is unbound on the VM → set -u aborts the whole deploy before Phase 7 → zero app layer. deploy-infra.sh ships a quoted-heredoc remote script (cat > … << 'EOF', runs on the VM via SSH); REPO_ROOT is assigned on the runner and never crosses. A bare bash "$REPO_ROOT/scripts/…" in that block dies REPO_ROOT: unbound variable AFTER Step-7 secrets but BEFORE the ApplicationSets apply — looks identical to Gotchas 12/13 (substrate green, no app pods) but the tell is deploy-infra-remote.sh: line NNN: <VAR>: unbound variable in the run log. Rule: any helper a remote-block needs must be scp'd to /tmp/ (alongside ensure-temporal-namespace.sh) and invoked as /tmp/<name>.sh, never via a runner path. Seen 2026-06-04: sync-app-webhook-secret.sh added in #1482 with $REPO_ROOT silently broke EVERY clean provision for a day (fixed: upload to /tmp + call /tmp/sync-app-webhook-secret.sh).
-
Phase 6 only applies the ExternalSecret leaves that EXIST on disk for that env — a missing per-(svc,env) leaf = that node never gets <node>-env-secrets → pod never Ready → Phase 9 /readyz reds the provision. Phase 6 globs the single SSOT tree nodes/<node>/k8s/external-secrets/<env>/ (every node carries its own leaf — operator, resy, scheduler-worker, canary, node-template; the pre-wizard operator-domain tree was purged). For an ESO-migrated node (consumes envFrom: <node>-env-secrets, not the deploy-infra bridge secret), if its leaf is absent for the target env, Phase 6 silently skips it (Applied N where N < node-count) and that pod boots without config. The trap: this only bites the env that's missing the leaf — candidate-a had all 5 leaves so it looked done; candidate-b shipped only 3 (scheduler-worker/canary/node-template), missing external-secrets/candidate-b/{operator,resy}/, so a clean provision reached Phase 6/7 fine but red'd at Phase 9 on operator+resy /readyz. Diagnose via the run's Applied N ExternalSecret manifest(s) count + the === ExternalSecrets sync state === dump (the missing nodes simply aren't listed). Fix = add the per-(svc,env) leaf (mirror an existing env, sed namespace: + dataFrom.extract.key:); it's committed → reproducible. Leaves are hand-authored per env today — a new env or node silently inherits this gap; the durable fix is to codegen the leaves from the catalog (same renderer class as render-node-overlays.sh). Seen 2026-06-04 (candidate-b clean-room).
-
Per-env node membership ≠ the global catalog — Phase 9 must verify only the nodes the env DEPLOYS, or a correctly-provisioned subset-env reds. NODE_TARGETS (from scripts/ci/lib/image-tags.sh) = every type:node in infra/catalog/* (operator, resy, canary, node-template). But an env deploys only what its ApplicationSet declares — equivalently, the nodes with an overlay under infra/k8s/overlays/<env>/ (the env-scoped set Phase 4b.5 already uses). A subset env (candidate-b = 3-node task.5094 OpenBao proof: canary/node-template/scheduler-worker, NO operator/resy) provisions DBs + secrets + ExternalSecret leaves for ALL catalog nodes (harmless over-provision), but its AppSet creates no operator/resy Argo app → no pod → Phase 9 (looping the global NODE_TARGETS) verifies operator/resy /readyz, fails after 5 min, and reds a provision whose declared nodes are all 1/1 + 200. The tell: Applied 5 ExternalSecret manifest(s) + ESO SecretSynced True for operator/resy, yet no operator/resy pod in kubectl get pods and no candidate-b-operator Argo app. Fix = scope Phase 9's loop to [[ -d infra/k8s/overlays/<env>/<node> ]] (skip undeclared). The deeper durable fix = make the per-env node set a single SSOT (the AppSet list) and derive NODE_TARGETS/DB/secret/verify scope from it, so a subset env doesn't over-provision. Seen 2026-06-04 (candidate-b); the old CANARY IS RED banner mislabels this (it fires for ANY node, not canary).
-
A provision run is 30–60 min — re-engaging on completion without going silent is its OWN failure mode, and every passive monitor tried here MISSED. Exact ledger + the rule. Mechanisms tried on the 2026-06-10 preview rebuild and why each went silent:
gh run watch <id> --exit-status via Bash run_in_background: true (the obvious "ping me when it finishes"). Fails: the background task is KILLED on every session resume / context-compaction boundary (the SessionStart hook fires repeatedly across a long wait). A killed task emits <task-notification status=killed> — NOT a completion — so the agent gets a "killed / Continue from where you left off" prompt, not the run result. Treated as terminal ⇒ silence.- Replying "No response requested" / any stand-down to that
status=killed notification. This was the silence. The killed-watcher notification is a RE-ARM trigger, never a terminal event, while the run is still pending.
ScheduleWakeup as a far-future BACKSTOP (1200s) behind the watcher. Roles inverted — the unreliable background watcher was treated as primary, the durable wakeup as a 20-min backstop. When the run went green early and the watcher had already been killed, nothing re-invoked the agent for up to 20 min ⇒ effective silence (the human poked first).ScheduleWakeup(270s) self-paced 5-min polls (early runs). These worked — durably re-invoked — but burn a turn every ~5 min for an hour.
RULE: ScheduleWakeup is the ONLY durable re-invoke for a provision/flight/promote wait — make it PRIMARY, never a backstop. (a) Match the delay to the phase you're gating on (a ~35-min provision → ~600–900s checks, not one 1200s shot). (b) A gh run watch background task is a nice-to-have fast path, never the sole signal — it WILL be killed across a resume. (c) Never end a turn while a run is pending without a ScheduleWakeup armed. (d) A status=killed/stopped task-notification means RE-ARM, not stand down. (e) Each wakeup's own prompt must say "re-arm before ending until <terminal condition>." Seen 2026-06-10: two silences on one preview rebuild — once from standing down on a killed watcher with no wakeup, once from a 20-min backstop that hadn't fired when the run went green.
