cluster-loop

name	cluster-loop
description	Operating manual for the local GitOps twin (kind + registry + Gitea + Flux) — how to drive it, how to read it, and the failure catalogue that cost real time to learn. Use when working on deploy/, clusters/, scripts/cluster.mjs, scripts/gitops.mjs, scripts/images.mjs, or when a Kustomization, Gateway, ImagePolicy or rollout is not doing what it says.

The cluster loop

pnpm cluster up (kind + registry) → pnpm gitops up (Gitea + flux bootstrap) → pnpm images build|push → Flux rolls it. pnpm gitops verify asserts the whole chain; pnpm gitops pull brings the bot's commits home.

Read this before touching a CRD field, marker or CLI flag

Every third-party contract is read from the thing that serves it, before first use (AGENTS.md §Hard rules). In this subsystem that means:

Question	Ask
CRD field exists / spelled how?	pnpm cluster exec -- "kubectl explain <kind>.spec --recursive"
Flag exists?	docker run --rm saasparilla-toolbox:latest "flux <cmd> --help"
Marker / template semantics?	fluxcd.io docs — fetch the page, do not recall it
Chart values path?	the chart's own values.yaml from its release tag

An hour was lost in M9 to one violation of this: {"$imagepolicy": "…:tag"} was placed on a Deployment's image: line, where it replaces the whole reference with a bare tag. The automation committed image: 0.1.1, the cluster tried to pull an image by that name, and the rollout stalled. :tag markers belong on a newTag: field in the kustomization's images: transformer; the plain form replaces a full reference.

Feedback-loop costs (measured 2026-08-13) — earn the slow ones

Step	Cost
pnpm deploy-lint / kubectl kustomize render	~1s
toolbox or app image build (cached / cold)	seconds / ~2–5m
pnpm cluster up (node Ready ~16s) / pnpm gitops up	~1m / ~2m
ImageRepository scan · Kustomization interval · apps timeout	1m · 10m · 15m

Spend from the top down: a wrong marker caught by the render costs 1s, the same mistake through the live loop costs a reconcile interval plus a stalled rollout. Two corollaries, both paid for this day:

• Hand-run a third-party flow's riskiest step ONCE before scripting around it. The whole Gitea script was built before flux bootstrap over HTTP was ever attempted; a toolbox one-liner would have said "basic auth cannot be sent over HTTP" in seconds, before the script existed to rebuild.
• Convergence waits run in the BACKGROUND with a deadline of the object's own timeout plus margin — a foreground until loop against a 10m interval blocks the session and times out anyway.

Memory: the Docker VM is ~7.75 GiB. M7's airbyte-abctl-control-plane holds ~3.8 GiB when running — docker stop it for cluster work; docker start restores it. The compose stack is another ~2 GB the cluster does not need.

Waiting is the expensive part — do not wait blind

• Lab intervals are 1m (sources, policies) and 10m (Kustomizations). A Kustomization with wait: true holds its whole timeout on a rollout that can never succeed, and ignores reconcile requests while in flight.
• To cancel a stuck reconcile: flux suspend kustomization X && flux resume kustomization X. This is the only fast way out — flux reconcile will just time out behind the in-flight one.
• flux get kustomizations mid-cascade shows dependency … not up to date. That is the ordering working, not a fault; layers converge in dependsOn order.

Two names for one registry, two planes for one DNS name

• Manifests say localhost:5000/... — resolved by containerd on the node via certs.d. A pod cannot use that name; localhost in a pod is the pod.
• Scanners (ImageRepository) say saasparilla-registry:5000 — pinned in CoreDNS by gitops up, alongside gitea.saasparilla.local.
• Therefore automation may rewrite tags, never hosts.
• Whether a pod resolves a Docker alias unaided depends on the Docker flavour (Docker Desktop hands the node a routable resolver; a container-local resolver cannot be reached from a pod). Never infer the pod plane from a toolbox curl — gitops verify resolves from inside a pod.

Failure catalogue (each one cost real time)

Symptom	Cause
basic auth cannot be sent over HTTP during bootstrap	Flux refuses basic auth on plain HTTP, no override. The git server must be HTTPS; --ca-file carries the CA into the flux-system secret
authentication required: Failed to authenticate user	a token rotation revoked the credential Flux holds. Never revoke; mint uniquely-named tokens
Programmed=False: No addresses have been assigned	kind has no load balancer; expose the data plane as NodePort via an EnvoyProxy CR on the GatewayClass parametersRef
Gateway/HTTPRoute reports True but is broken	Gateway API status is left behind when a controller stops managing an object. Compare status against the current spec
invalid literal for int(): 'tcp://10.96…'	Kubernetes injects <SERVICE>_PORT link vars into every pod. Set enableServiceLinks: false on workloads taking envFrom
template uses removed '.Updated' field	use .Changed.Objects; the controller refuses rather than committing garbage
field is immutable on a Job	image automation rewrote its tag. Migrations belong in an initContainer, not a Job

Before the live loop, prove it offline

pnpm deploy-lint runs with no cluster and enumerates every kustomize directory, the cluster layer's dependsOn DAG, image pins (including initContainers), and enableServiceLinks. A mechanism with a rewrite step — image automation, a kustomize transformer — is provable offline too: edit the manifest by hand to the shape the automation would produce and run kubectl kustomize deploy/base. That check costs seconds; the live version costs a rollout.