| name | cluster-loop |
| description | Operating manual for the local GitOps twin (kind + registry + Gitea + Flux) — how to drive it, how to read it, and the failure catalogue that cost real time to learn. Use when working on deploy/, clusters/, scripts/cluster.mjs, scripts/gitops.mjs, scripts/images.mjs, or when a Kustomization, Gateway, ImagePolicy or rollout is not doing what it says. |
The cluster loop
pnpm cluster up (kind + registry) → pnpm gitops up (Gitea + flux bootstrap) → pnpm images build|push → Flux rolls it. pnpm gitops verify
asserts the whole chain; pnpm gitops pull brings the bot's commits home.
Read this before touching a CRD field, marker or CLI flag
Every third-party contract is read from the thing that serves it, before
first use (AGENTS.md §Hard rules). In this subsystem that means:
| Question | Ask |
|---|
| CRD field exists / spelled how? | pnpm cluster exec -- "kubectl explain <kind>.spec --recursive" |
| Flag exists? | docker run --rm saasparilla-toolbox:latest "flux <cmd> --help" |
| Marker / template semantics? | fluxcd.io docs — fetch the page, do not recall it |
| Chart values path? | the chart's own values.yaml from its release tag |
An hour was lost in M9 to one violation of this: {"$imagepolicy": "…:tag"} was placed on a Deployment's image: line, where it replaces the
whole reference with a bare tag. The automation committed image: 0.1.1,
the cluster tried to pull an image by that name, and the rollout stalled.
:tag markers belong on a newTag: field in the kustomization's images:
transformer; the plain form replaces a full reference.
Feedback-loop costs (measured 2026-08-13) — earn the slow ones
| Step | Cost |
|---|
pnpm deploy-lint / kubectl kustomize render | ~1s |
| toolbox or app image build (cached / cold) | seconds / ~2–5m |
pnpm cluster up (node Ready ~16s) / pnpm gitops up | ~1m / ~2m |
| ImageRepository scan · Kustomization interval · apps timeout | 1m · 10m · 15m |
Spend from the top down: a wrong marker caught by the render costs 1s, the
same mistake through the live loop costs a reconcile interval plus a stalled
rollout. Two corollaries, both paid for this day:
- Hand-run a third-party flow's riskiest step ONCE before scripting around
it. The whole Gitea script was built before
flux bootstrap over HTTP was
ever attempted; a toolbox one-liner would have said "basic auth cannot be
sent over HTTP" in seconds, before the script existed to rebuild.
- Convergence waits run in the BACKGROUND with a deadline of the object's
own timeout plus margin — a foreground
until loop against a 10m interval
blocks the session and times out anyway.
Memory: the Docker VM is ~7.75 GiB. M7's airbyte-abctl-control-plane holds
~3.8 GiB when running — docker stop it for cluster work; docker start
restores it. The compose stack is another ~2 GB the cluster does not need.
Waiting is the expensive part — do not wait blind
- Lab intervals are 1m (sources, policies) and 10m (Kustomizations).
A Kustomization with
wait: true holds its whole timeout on a rollout that
can never succeed, and ignores reconcile requests while in flight.
- To cancel a stuck reconcile:
flux suspend kustomization X && flux resume kustomization X. This is the only fast way out — flux reconcile will just
time out behind the in-flight one.
flux get kustomizations mid-cascade shows dependency … not up to date.
That is the ordering working, not a fault; layers converge in dependsOn order.
Two names for one registry, two planes for one DNS name
- Manifests say
localhost:5000/... — resolved by containerd on the node
via certs.d. A pod cannot use that name; localhost in a pod is the pod.
- Scanners (ImageRepository) say
saasparilla-registry:5000 — pinned in
CoreDNS by gitops up, alongside gitea.saasparilla.local.
- Therefore automation may rewrite tags, never hosts.
- Whether a pod resolves a Docker alias unaided depends on the Docker
flavour (Docker Desktop hands the node a routable resolver; a
container-local resolver cannot be reached from a pod). Never infer the pod
plane from a toolbox
curl — gitops verify resolves from inside a pod.
Failure catalogue (each one cost real time)
| Symptom | Cause |
|---|
basic auth cannot be sent over HTTP during bootstrap | Flux refuses basic auth on plain HTTP, no override. The git server must be HTTPS; --ca-file carries the CA into the flux-system secret |
authentication required: Failed to authenticate user | a token rotation revoked the credential Flux holds. Never revoke; mint uniquely-named tokens |
Programmed=False: No addresses have been assigned | kind has no load balancer; expose the data plane as NodePort via an EnvoyProxy CR on the GatewayClass parametersRef |
Gateway/HTTPRoute reports True but is broken | Gateway API status is left behind when a controller stops managing an object. Compare status against the current spec |
invalid literal for int(): 'tcp://10.96…' | Kubernetes injects <SERVICE>_PORT link vars into every pod. Set enableServiceLinks: false on workloads taking envFrom |
template uses removed '.Updated' field | use .Changed.Objects; the controller refuses rather than committing garbage |
field is immutable on a Job | image automation rewrote its tag. Migrations belong in an initContainer, not a Job |
Before the live loop, prove it offline
pnpm deploy-lint runs with no cluster and enumerates every kustomize
directory, the cluster layer's dependsOn DAG, image pins (including
initContainers), and enableServiceLinks. A mechanism with a rewrite step —
image automation, a kustomize transformer — is provable offline too: edit the
manifest by hand to the shape the automation would produce and run
kubectl kustomize deploy/base. That check costs seconds; the live version
costs a rollout.
