loom-argocd

# Argo CD GitOps Continuous Delivery ## Overview Declarative GitOps CD for Kubernetes: Git is the source of truth, a controller reconciles cluster state toward it. Vocabulary you must be exact about: **target state** (Git), **live state** (cluster), **sync** (make live match target), **OutOfSync** (they differ), **health** (per-resource readiness, distinct from sync), **refresh** (recompare Git vs live), **AppProject** (RBAC/resource boundary grouping Applications). Two facts that trip up most newcomers, stated up front: - **OutOfSync ≠ unhealthy, and Healthy ≠ InSync.** They are orthogonal axes. A perpetually-OutOfSync-but-Healthy app almost always means a controller mutates a field after apply — fix with `ignoreDifferences` (see Expert Practices), not by re-syncing. - **Argo CD only inflates manifests** (`helm template`, `kustomize build`) — it never runs `helm install`. `helm ls`/`helm rollback` show nothing. ## When Argo CD vs Flux Both are CNCF-graduated GitOps controllers; the choice is architectural, not feature-parity. | Concern | Argo CD | Flux CD | | --- | --- | --- | | Ordering | `argocd.argoproj.io/sync-wave` annotations *within* one Application; cross-Application needs ApplicationSet Progressive Sync | `dependsOn` between Kustomizations/HelmReleases + `healthChecks` gate the next | | Composition | App-of-apps: one root Application recursing into child Applications | Kustomization tree: a Kustomization applies more Kustomizations | | Drift correction | Opt-in `syncPolicy.automated.selfHeal` reverts live drift; `prune` deletes Git-removed resources | Continuous reconciliation always re-applies desired state; `prune: true` GCs by `.status.inventory` | | Fan-out | ApplicationSet generators (cluster/git/matrix/PR/SCM) | No native generator; per-tenant Kustomizations + image automation | | Image updates | Not built-in (separate Argo CD Image Updater) | First-class ImageRepository/ImagePolicy/ImageUpdateAutomation, commits back to Git | | Interface | Web UI-centric (topology view, manual sync buttons, RBAC console) | CLI/CRD-centric (`flux` CLI, no first-party UI) | | Multi-cluster | One control plane syncs many clusters | Typically one Flux per cluster pulling its own path | Rule of thumb: **Argo CD** when operators want a visual sync/health console and generator-driven multi-cluster fan-out; **Flux** for a lean controller set, Git-native image automation, and dependency ordering expressed as CRDs. They coexist (Argo for app-team UI, Flux for platform automation). ## Install & Setup ```bash kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml # Production: swap install.yaml -> ha/install.yaml (redis-ha, controller replicas) kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d kubectl port-forward svc/argocd-server -n argocd 8080:443 argocd login localhost:8080 argocd account update-password argocd cluster add my-cluster-context # register an external cluster argocd repo add https://github.com/org/repo.git --username u --password $TOKEN ``` ## Repository Structure ```text gitops-repo/ ├── apps/{base,overlays/{dev,staging,production}}/ # Kustomize bases + overlays ├── charts/myapp/ # Helm charts (if used) ├── argocd/{projects,applications,applicationsets}/ # Argo CRDs └── bootstrap/root-app.yaml # app-of-apps entrypoint ``` | Separation | Pros | Cons | | --- | --- | --- | | Mono-repo | Simple, atomic cross-cutting changes | Coarse access control | | Repo-per-env | Clear promotion path, security boundaries | Config duplication | | Repo-per-team | Team autonomy/ownership | Cross-team coordination | ## Application Manifests Basic Application (annotated — the fields that matter): ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: myapp namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io # cascade-delete managed resources (see Deletion gotcha) spec: project: default source: repoURL: https://github.com/org/repo.git targetRevision: HEAD # branch, tag, or immutable SHA (prefer SHA for prod) path: apps/production/myapp destination: server: https://kubernetes.default.svc namespace: myapp-production syncPolicy: automated: prune: true # delete resources removed from Git selfHeal: true # revert manual cluster edits (fights kubectl) allowEmpty: false # refuse to sync an empty source (prevents mass-delete) syncOptions: - CreateNamespace=true - PrunePropagationPolicy=foreground - PruneLast=true # prune only after new resources are Healthy retry: limit: 5 backoff: { duration: 5s, factor: 2, maxDuration: 3m } ``` Source-type deltas (everything else identical to above). **Helm** — Argo runs `helm template`; `valueFiles` are last-wins (see Helm gotcha): ```yaml source: path: charts/myapp helm: valueFiles: [values.yaml, values-production.yaml] # last file wins values: | # inline, higher precedence replicaCount: 3 parameters: # highest precedence - { name: image.tag, value: v1.2.3 } releaseName: myapp # ⚠ overriding this changes app.kubernetes.io/instance label ``` **Kustomize** — Argo runs `kustomize build`: ```yaml source: path: apps/overlays/production kustomize: namePrefix: prod- images: - { name: myapp, newName: myregistry.io/myapp, newTag: v1.2.3 } replicas: - { name: myapp-deployment, count: 3 } ``` ## ApplicationSets One ApplicationSet templates many Applications from a **generator**. Use for multi-cluster fan-out, multi-tenant, or discovering apps from repo structure. | Generator | Use case | | --- | --- | | **cluster** | Same app to every registered cluster (matched by label) | | **git (directories)** | One app per directory in a monorepo | | **git (files)** | One app per config file (JSON/YAML params) | | **list** | Static parameter list (tenant definitions) | | **matrix** | Cross-product of two generators (apps × clusters/envs) | | **pullRequest** | Ephemeral preview env per PR | | **scmProvider** | Discover repos org-wide from GitHub/GitLab | Cluster generator (fan-out to labelled clusters): ```yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: { name: cluster-apps, namespace: argocd } spec: goTemplate: true goTemplateOptions: [missingkey=error] # ⚠ else typos render blank targets — see gotcha generators: - clusters: selector: { matchLabels: { env: production } } template: metadata: { name: '{{.name}}-myapp' } spec: project: platform-team # ⚠ hard-code, never template (privilege escalation) source: { repoURL: https://github.com/org/repo.git, targetRevision: main, path: apps/production/myapp } destination: { server: '{{.server}}', namespace: myapp } syncPolicy: { automated: { prune: true, selfHeal: true }, syncOptions: [CreateNamespace=true] } ``` Matrix (apps × environments — the common multi-env pattern): ```yaml spec: generators: - matrix: generators: - git: { repoURL: https://github.com/org/apps.git, revision: HEAD, directories: [{ path: apps/* }] } - list: elements: - { env: dev, cluster: https://dev.example.com, replicas: "1" } - { env: production, cluster: https://prod.example.com, replicas: "3" } template: metadata: { name: '{{path.basename}}-{{env}}' } spec: project: default source: repoURL: https://github.com/org/apps.git targetRevision: HEAD path: '{{path}}' helm: { parameters: [{ name: replicaCount, value: '{{replicas}}' }] } destination: { server: '{{cluster}}', namespace: '{{path.basename}}-{{env}}' } ``` PR previews use `generators: [{ pullRequest: { github: { owner, repo, tokenRef }, labels: [preview] } }]` with `targetRevision: '{{head_sha}}'` and `namespace: myapp-pr-{{number}}`. SCM discovery uses `scmProvider.github` with `filters: [{ repositoryMatch: '.*-service$', pathsExist: [k8s/production] }]`. ## Sync Policy & Strategies | Strategy | Use | Risk | | --- | --- | --- | | Automated + selfHeal | Non-prod | Low | | Automated, no selfHeal | Staging (manual drift control) | Medium | | Manual | Production | High | | Sync windows | Business-hours restriction | Medium | | Progressive (Argo Rollouts) | Gradual prod rollout | Low | Full `syncOptions` reference (add to `syncPolicy.syncOptions`): - `CreateNamespace=true` — create the destination namespace. - `ServerSideApply=true` — SSA field ownership; required for large CRDs and self-managed Argo (see gotcha). - `PruneLast=true` — prune after new resources Healthy. - `PrunePropagationPolicy=foreground|background|orphan`. - `RespectIgnoreDifferences=true` — honor `ignoreDifferences` *during sync*, not just diffing (see gotcha). - `ApplyOutOfSyncOnly=true` — skip already-synced resources (faster large syncs; do NOT use right after a tracking-method change). - `Validate=false` — skip `kubectl` schema validation. - `Replace=true` — `kubectl replace` instead of apply (⚠ overrides ServerSideApply). Sync windows (time-gated deploys, defined on AppProject): ```yaml kind: AppProject spec: syncWindows: - { kind: allow, schedule: "0 9 * * 1-5", duration: 8h, applications: ["*"] } # business hours - { kind: deny, schedule: "0 12 * * *", duration: 2h, applications: ["*"] } # peak-traffic freeze - { kind: allow, schedule: "* * * * *", duration: 1h, manualSync: true, applications: [critical-app] } ``` `ignoreDifferences` on the Application stops sync loops from controller-mutated fields (pair with `RespectIgnoreDifferences=true`): ```yaml spec: ignoreDifferences: - { group: apps, kind: Deployment, jsonPointers: [/spec/replicas] } # HPA-managed - group: "" kind: Service jqPathExpressions: [".spec.ports[] | select(.nodePort != null) | .nodePort"] ``` > There is **no `syncWaves` field** in the Application CRD. Argo tolerates unknown fields, so such a block is silently ignored. Wave ordering is *only* the `argocd.argoproj.io/sync-wave` annotation on individual resources. For blue-green/canary with traffic shifting, use **Argo Rollouts** (separate `Rollout` CRD), not the Application spec. ## App of Apps Pattern A root Application whose source is a directory of child Application manifests (`directory.recurse: true`), so one bootstrap sync creates the whole tree. ```text root-app ├── infrastructure (sync-wave 0): cert-manager, ingress-nginx, external-dns ├── platform (sync-wave 1): monitoring, logging, security └── applications (sync-wave 2): app1, app2, app3 ``` ```yaml kind: Application metadata: name: root-app namespace: argocd # ⚠ resources-finalizer intentionally OMITTED — with it, one `kubectl delete # application root-app` cascades through every child and wipes the environment. spec: project: default source: { repoURL: https://github.com/org/gitops.git, targetRevision: HEAD, path: argocd/applications, directory: { recurse: true } } destination: { server: https://kubernetes.default.svc, namespace: argocd } syncPolicy: { automated: { prune: true, selfHeal: true }, syncOptions: [CreateNamespace=true] } ``` > **Restore the Application health check or wave ordering is a lie.** The built-in health assessment for the `argoproj.io/Application` CRD was **removed in Argo CD 1.8**. Without it the parent marks a child "done" the instant the Application *object* is applied — not when its workloads are Healthy — so each wave advances before the previous layer is Ready. Restore it in `argocd-cm` (see Expert Practices). Even restored, child-Application sync-waves are reliable only at *initial bootstrap*; for ongoing ordered multi-app rollouts use **ApplicationSet Progressive Syncs (RollingSync)**. ## Sync Waves & Hooks Waves (`argocd.argoproj.io/sync-wave`, lower syncs first; default 0) order resources within one Application; Argo applies a wave, waits for Healthy, then advances. Typical: Namespace `"0"` → ConfigMap `"1"` → Deployment `"2"` → Service `"3"`. Hooks are Jobs annotated with `argocd.argoproj.io/hook`: | Hook | Runs | | --- | --- | | `PreSync` | Before sync (DB migrations) | | `Sync` | During, with the main wave | | `PostSync` | After all Healthy (smoke tests) | | `SyncFail` | On sync failure (alert/rollback) | | `PreDelete` | Before Application deletion | `argocd.argoproj.io/hook-delete-policy`: `HookSucceeded` | `HookFailed` | `BeforeHookCreation`. Hooks also honor sync-wave annotations. ```yaml kind: Job metadata: name: db-migration annotations: argocd.argoproj.io/hook: PreSync argocd.argoproj.io/hook-delete-policy: HookSucceeded argocd.argoproj.io/sync-wave: "1" spec: backoffLimit: 3 template: { spec: { restartPolicy: Never, containers: [{ name: migrate, image: myapp:migrations, command: ["./migrate.sh"] }] } } ``` ## Health Checks & Resource Customizations Custom health for CRDs Argo doesn't understand (Lua in `argocd-cm`): ```yaml kind: ConfigMap metadata: { name: argocd-cm, namespace: argocd } data: resource.customizations.health.cert-manager.io_Certificate: | hs = {} if obj.status ~= nil and obj.status.conditions ~= nil then for _, c in ipairs(obj.status.conditions) do if c.type == "Ready" and c.status == "True" then return { status = "Healthy", message = c.message } end if c.type == "Ready" and c.status == "False" then return { status = "Degraded", message = c.message } end end end return { status = "Progressing", message = "Waiting for certificate" } # Ignore differences globally (e.g. fields owned by another controller) resource.customizations.ignoreDifferences.apps_Deployment: | jsonPointers: [/spec/replicas] # Resource tracking: 3.0 default is `annotation`. `annotation+label` writes both # for migration compatibility. ⚠ Switching mid-lifecycle can orphan a resource if