Declarative GitOps CD: a set of specialized controllers continuously reconcile cluster state toward Git. Flux is CRD-centric with no first-party UI — you drive it with the flux CLI and YAML. The controllers, and the CRDs each owns:
First-class ImageRepository/ImagePolicy/ImageUpdateAutomation, commits back to Git
Not built-in (separate Argo CD Image Updater)
Interface
CLI/CRD-centric (flux CLI, no first-party UI)
Web UI-centric (topology, manual sync buttons)
Multi-cluster
Typically one Flux per cluster pulling its own path
One control plane syncs many clusters
Rule of thumb: Flux for a lean controller set, Git-native image automation, and dependency ordering expressed as CRDs; Argo CD when operators want a visual sync/health console and generator-driven multi-cluster fan-out. They coexist.
Bootstrap (not flux install) writes the components into Git so they are version-controlled and self-managed — required to patch controller args via Kustomize (concurrency, lockdown flags). Validate manifests pre-commit with kubectl apply --dry-run=server -f clusters/production/.
Repository Structure
├── clusters/{production,staging}/
│ ├── flux-system/ # bootstrapped components (managed by Flux itself)
│ ├── infrastructure.yaml # sources + Kustomizations for infra
│ └── apps.yaml # sources + Kustomizations for apps
├── infrastructure/{base,overlays/{production,staging}}/ # ingress, cert-manager, ...
└── apps/{base,overlays/{production,staging}}/
apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata: { name:apps, namespace:flux-system }
spec:interval:10m0sretryInterval:2m0s# ⚠ defaults to interval — set independently (see gotcha)dependsOn: [{ name:infrastructure }]
sourceRef: { kind:GitRepository, name:flux-system }
path:./apps/productionprune:truewait:true# ⚠ health-checks ALL resources; silently ignores .healthCheckstimeout:5m0spostBuild:substitute: { cluster_name:production }
substituteFrom:- { kind:ConfigMap, name:cluster-vars }
postBuild.substitute/substituteFrom replace ${var} tokens in the built manifests (variable names must match ^[_[:alpha:]][_[:alpha:][:digit:]]*$ — hyphens/dots silently skip). The referenced ConfigMap/Secret:
kind:ConfigMapmetadata:name:cluster-varsnamespace:flux-systemlabels: { reconcile.fluxcd.io/watch:Enabled } # ⚠ else edits ignored until next tick (see gotcha)data: { cluster_name:production, domain:example.com }
Dependency & Ordering
Flux orders reconciliation with dependsOn (a Kustomization/HelmRelease waits for the named object to become Ready) combined with healthChecks/wait. This is Flux's answer to Argo sync-waves and app-of-apps, expressed as a CRD graph.
CRD Kustomizations should set prune: false so a transient source error never GCs your CRDs (and every CR with them). Cross-namespace dependsOn names the namespace: dependsOn: [{ name: shared-ingress, namespace: flux-system }].
Helm Integration
HelmRepository (or OCIRepository) provides charts; HelmRelease installs them. helm-controller runs real Helm (unlike Argo's helm template), so helm history/rollback work.
Prefer chartRef + OCIRepository over chart.spec for shared/pinned/signed charts (see Expert Practices). Private HelmRepository uses secretRef to a Secret with stringData.{username,password}.
Secret Management (SOPS)
Flux decrypts SOPS-encrypted manifests inline during Kustomization apply.
# .sops.yaml — encrypt only the data fields, per pathcreation_rules:-path_regex:.*/production/.*\.yamlencrypted_regex:^(data|stringData)$age:age1ql3z...# comma-separate multiple recipients for team access---# Kustomization decrypts:spec:decryption:provider:sopssecretRef: { name:sops-age }
Alternatives: External Secrets Operator (pull from AWS SM/Vault/GCP via SecretStore+ExternalSecret) — preferred for cloud secret managers; Sealed Secrets — Kubernetes-native one-way encryption.
Image Automation
Three resources form the loop: ImageRepository (scans a registry) → ImagePolicy (selects a tag) → ImageUpdateAutomation (commits the new tag to Git). Manifests carry a marker comment the automation rewrites.
apiVersion:image.toolkit.fluxcd.io/v1kind:ImageRepositorymetadata: { name:my-app, namespace:flux-system }
spec:image:ghcr.io/org/my-appinterval:5m0sprovider:aws# ⚠ prefer workload identity over secretRef (see Security)---apiVersion:image.toolkit.fluxcd.io/v1kind:ImagePolicymetadata: { name:my-app, namespace:flux-system }
spec:imageRepositoryRef: { name:my-app }
policy: { semver: { range:1.0.x } } # or alphabetical/numerical (below)filterTags:# ⚠ non-matching tags are dropped, no fallbackpattern:"^main-[a-f0-9]+-(?P<ts>[0-9]{10})$"extract:"$ts"---apiVersion:image.toolkit.fluxcd.io/v1kind:ImageUpdateAutomationmetadata: { name:my-app, namespace:flux-system }
spec:interval:1m0ssourceRef: { kind:GitRepository, name:flux-system }
git:checkout: { ref: { branch:main } }
push: { branch:image-updates } # omit for direct commit; set for PR-based flowcommit:author: { email:fluxcdbot@users.noreply.github.com, name:fluxcdbot }
messageTemplate:"Automated image update [ci skip]"update: { path:./apps/production, strategy:Setters }
# Deployment marker the automation rewrites:image:ghcr.io/org/my-app:1.0.0# {"$imagepolicy": "flux-system:my-app"}
Policy types: semver (releases, 1.0.x/>=1.0.0), alphabetical (branch tags via filterTags), numerical (build numbers). Strategy: enable automation in dev/staging with direct commit; use push.branch (PR review) for production.
Multi-Tenancy
RBAC alone does NOT make Flux multi-tenant. A default install lets a tenant reference Sources/Secrets in other namespaces, pull arbitrary remote Kustomize bases, and (if it omits serviceAccountName) reconcile with the controller's cluster-wide identity. Three controller flags, applied as bootstrap Kustomize patches in clusters/<env>/flux-system/, close these vectors and are mandatory:
--default-service-account=default (kustomize/helm) — resources without spec.serviceAccountName fall back to the powerless namespace default SA instead of the controller identity.
With these set, every tenant Kustomization/HelmRelease MUST declare spec.serviceAccountName, bound via a namespace-scoped RoleBinding to a custom Role or the built-in admin ClusterRole — never a ClusterRoleBinding, never cluster-admin.
Hub-and-spoke: one Flux reconciles remote clusters via kubeConfig.secretRef, or (more common) one Flux per cluster pulling its own clusters/<env>/ path. Per-cluster variance is expressed with postBuild.substitute, not branching.
Reduce load with higher interval on stable resources, a higher retryInterval, and GitRepository.spec.ignore to shrink clones.
Expert Practices: Idioms, Anti-Patterns & Gotchas
The patterns above get a cluster running; this captures the non-obvious behavior that separates a working install from a correct one. Most are silent failures — no error, just wrong behavior.
Currency: stable API versions
Use stable APIs; betas are removed in Flux 2.7+ with no compatibility shim. After a CRD upgrade any beta apiVersion is rejected:
HelmRelease → helm.toolkit.fluxcd.io/v2 (stable since 2.3).
ImageRepository/ImagePolicy/ImageUpdateAutomation → image.toolkit.fluxcd.io/v1 (promoted in 2.7, Sep 2025, which removed the betas).
The v2 HelmRelease API dropped three fields with no in-place equivalent: .spec.chart.spec.valuesFile (use plural valuesFiles), and postRenderers.kustomize.patchesJson6902/patchesStrategicMerge (both unified into patches). Rewrite mechanically with flux migrate before upgrading controllers.
Idioms
Prefer chartRef + OCIRepository over chart.spec for shared/pinned/signed charts.chart.spec creates a hidden managed HelmChart per HelmRelease, pinnable only by version. chartRef points at an existing OCIRepository/HelmChart so multiple releases share one source, supports digest pinning (immutable deploys) and Cosign/notation verification. Mutually exclusive with chart.spec; HelmRepository type: oci is in maintenance mode. Switching an existing release to chartRef is a Helm upgrade (not reinstall) and GCs the old HelmChart.
Set retryInterval independently from interval. Orthogonal timers: interval is steady-state drift detection (min 60s), retryInterval is failure recovery, defaulting to interval when unset. An interval: 1h resource waits a full hour to retry a transient failure unless you lower retryInterval.
Label referenced ConfigMaps/Secrets reconcile.fluxcd.io/watch: Enabled. By default Flux re-reconciles only on the interval tick, so editing a ConfigMap in postBuild.substituteFrom or a Secret in valuesFrom isn't picked up until the next scheduled reconcile (possibly hours). The label (Flux 2.7) makes the controller watch and reconcile immediately.
Gotchas (silent failures)
wait: true silently ignores healthChecks — they are mutually exclusive. With wait: true the Kustomization health-checks all reconciled resources and .spec.healthChecks is ignored — setting both gives a false sense of targeted gating. To gate on a named subset, leave wait unset and use healthChecks alone.
postBuild substitution traps. (a) Runs only if at least one substitute/substituteFrom is defined — otherwise ${var:=default} passes through literally. (b) Var names must match ^[_[:alpha:]][_[:alpha:][:digit:]]*$ — a hyphen/dot means silent skip. (c) An undefined ${VAR} with no default becomes an empty string, so a typo ${cluster_rgion} silently corrupts a URL. (d) Quote numbers/booleans to avoid YAML coercion. Harden with --feature-gates=StrictPostBuildSubstitutions=true and validate via flux build kustomization --strict-substitute.
Renaming a prune: true Kustomization (or moving resources between two) deletes its workloads. Flux tracks owned resources in .status.inventory by name+namespace; rename the object and the whole inventory is GC'd then recreated — a momentary outage. Safe procedure: prune: false, reconcile, verify the renamed object is Ready and owns the resources, then re-enable prune. Per-resource opt-out: kustomize.toolkit.fluxcd.io/prune: disabled.
HelmRelease drift detection is Disabled by default. helm-controller does NOT correct out-of-band kubectl edits unless spec.driftDetection.mode is set — divergence is silent until the next Helm action. warn logs via events; enabled corrects via server-side dry-run apply. Companion trap: once enabled, any legitimate mutator (HPA on /spec/replicas, VPA, cert-manager CA) gets reverted every cycle — add driftDetection.ignore paths. Start with warn to discover them.
HelmRelease valuesFrom with targetPath has the HIGHEST precedence — above inline spec.values.valuesFrom entries merge left-to-right, then inline values overwrites — BUT a valuesFrom entry with targetPath overwrites everything before it, including inline values. (Also: deleting a ConfigMap/Secret referenced in valuesFrom changes inputs and triggers a Helm upgrade.)
upgrade.remediation defaults are asymmetric.install.remediation.remediateLastFailure defaults false; upgrade.remediation.remediateLastFailure defaults false UNLESS .retries > 0, when it flips to true — so merely adding an upgrade retry count silently enables last-failure rollback. Be explicit, pair with cleanupOnFail, avoid retries: -1 on a broken chart.
HelmRelease release name is silently SHA-256-truncated past 53 chars. Flux composes [<targetNamespace>-]<HelmRelease.name>; over Helm's 53-char DNS-label limit it becomes first-40-chars + dash + first-12 of a SHA-256 hash. helm list/history then won't show the expected name. Set spec.releaseName explicitly when the composed name could approach 53 chars.
kubectl rollout restart on a Flux-managed resource churns. It adds restartedAt; the next reconcile removes it (not in Git) and redeploys — a loop. Use the Flux field manager: kubectl rollout restart deploy/my-app -n apps --field-manager=flux-client-side-apply. (Any kubectl edit is likewise reverted — intentional drift correction.)
filterTags.extract drops non-matching tags entirely — no fallback.pattern selects candidate tags; extract supplies a derived sort value (e.g. captured timestamp) — it does not rename or fall back. A wrong regex yields zero candidates and "no latest image", not all-tags. Companion: digestReflectionPolicy: Always requires an interval; IfNotPresent/Never forbid it.
Image automation needs a read-write deploy key; re-bootstrapping does NOT rotate it.flux bootstrap creates a read-only key by default, so image-automation-controller silently fails to push without --read-write-key. Re-running bootstrap with the flag does NOT overwrite the existing flux-system Secret — delete it first, then re-bootstrap:
kubectl delete secret flux-system -n flux-system
flux bootstrap github --read-write-key ... # Secret recreated with a write key
Also: ImageUpdateAutomation evaluates only ImagePolicy objects in its own namespace — cross-namespace policy refs are unsupported.
The two Kustomization kinds are different objects.kustomization.kustomize.toolkit.fluxcd.io is the Flux CR (a reconciliation unit sourcing from a GitRepository, optionally applying an overlay); kustomization.kustomize.config.k8s.io is the native kustomize file. The Flux CR's spec.path points at a directory containing the config-kind kustomization.yaml — it orchestrates, not replaces. Native fields (resources, patches, configMapGenerator) belong in the file, never the Flux CR spec.
Anti-patterns
Never bind a tenant reconciler to cluster-admin (or any ClusterRoleBinding) — it defeats namespace isolation. Use a namespace-scoped RoleBinding to a custom Role or the built-in admin ClusterRole, plus the lockdown flags.
force: true is a temporary escape hatch, not a setting. It makes the controller delete-then-recreate resources when an immutable-field patch fails — bypassing Kubernetes immutability guards for EVERY managed resource. Left on, it removes protection against accidental data loss on stateful workloads. Prefer the per-resource annotation kustomize.toolkit.fluxcd.io/force: enabled on the one object, then remove it.
Security
Multi-tenancy is not enforced by default — --no-cross-namespace-refs, --no-remote-bases, --default-service-account are mandatory; omitting any one leaves a privilege-escalation path RBAC alone does not close (see Multi-Tenancy).
Ban Kustomize remote bases in production. Bases pointing at external URLs are fetched at reconcile time over HTTPS, outside Flux's artifact pipeline: no crypto verification, no caching (refetched every cycle), no immutability, absent from source history — a supply-chain risk. Disable with --no-remote-bases=true; replace with a Flux OCIRepository/GitRepository pinned by digest.
Use workload identity instead of static credential Secrets. Flux 2.7 completed object-level Kubernetes Workload Identity for all cloud-authenticating APIs (GitRepository, OCIRepository, ImageRepository, Bucket, Kustomization, HelmRelease, Provider) on AWS (EKS IRSA), Azure (AKS WI), GCP (GKE WI). Set .spec.provider: aws|azure|gcp so the controller fetches short-lived OIDC tokens instead of reading a static Secret — no rotation burden, smaller blast radius.
Decision Points
Choice
Take A when
Take B when
GitRepository vs HelmRepository
custom manifests / Kustomize / charts in Git
public/private Helm chart repo
Kustomization vs HelmRelease
raw manifests, overlays, ConfigMaps/Secrets
packaged charts with values
Image automation
direct commit (dev/staging)
push.branch PR review (prod), or disabled (manual gate)
