GitOps Patterns
Best practices for operating infrastructure and applications through Git as the single source of truth. Covers ArgoCD, Flux, Argo Rollouts, environment promotion, secret management, and progressive delivery.
GitOps Principles
GitOps extends Infrastructure as Code by adding a reconciliation loop that continuously ensures the live system matches the declared state in Git.
| Principle | Description | Implementation |
|---|
| Declarative | Entire system described declaratively | Kubernetes manifests, Helm charts, Kustomize overlays |
| Versioned and immutable | Desired state stored in Git | Git commits as audit trail, tags for releases |
| Pulled automatically | Software agents pull desired state | ArgoCD, Flux controllers watch Git repos |
| Continuously reconciled | Agents correct drift automatically | Reconciliation loop detects and fixes divergence |
| GitOps vs Traditional CI/CD | GitOps | Traditional CI/CD |
|---|
| Deployment trigger | Git commit (pull-based) | Pipeline step (push-based) |
| Source of truth | Git repository | Pipeline state / manual |
| Drift handling | Auto-corrected | Undetected until next deploy |
| Rollback method | git revert | Re-run pipeline with old version |
| Audit trail | Git log | Pipeline logs (often ephemeral) |
| Credential exposure | Cluster-only (no CI secrets) | CI system has deploy credentials |
Repository Structure
Separate application source code from deployment manifests. This is the most critical architectural decision in GitOps.
# Deployment repo (gitops-deploy) -- separate from application source
gitops-deploy/
apps/
backend/
base/
deployment.yaml
service.yaml
kustomization.yaml
overlays/
dev/
kustomization.yaml # image: backend:dev-abc1234
staging/
kustomization.yaml # image: backend:staging-def5678
prod/
kustomization.yaml # image: backend:v1.2.3
patches/
replicas.yaml
hpa.yaml
frontend/
base/
overlays/
infrastructure/
cert-manager/
ingress-nginx/
monitoring/
clusters/
dev/
kustomization.yaml # References apps/*/overlays/dev
staging/
kustomization.yaml
prod/
kustomization.yaml
| Structure Pattern | When to Use | Trade-offs |
|---|
| Monorepo (app + deploy) | Small teams, single app | Simple but couples build and deploy |
| Separate repos | Multiple teams, compliance needs | Clean separation, more repos to manage |
| Branch-per-env | Simple promotion model | Risk of branch divergence, merge conflicts |
| Directory-per-env | Kustomize overlays | Recommended: explicit, reviewable, no branch drift |
ArgoCD Application Manifest
ArgoCD watches Git repositories and synchronizes Kubernetes cluster state to match.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: backend-prod
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: production
source:
repoURL: https://github.com/myorg/gitops-deploy.git
targetRevision: main
path: apps/backend/overlays/prod
kustomize:
images:
- backend=ghcr.io/myorg/backend:v1.2.3
destination:
server: https://kubernetes.default.svc
namespace: backend
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
retry:
limit: 3
ArgoCD ApplicationSet for Multi-Environment
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: backend
namespace: argocd
spec:
generators:
- list:
elements:
- env: dev
cluster: https://dev-cluster.example.com
autoSync: true
- env: staging
cluster: https://staging-cluster.example.com
autoSync: true
- env: prod
cluster: https://prod-cluster.example.com
autoSync: false
template:
metadata:
name: 'backend-{{env}}'
spec:
project: '{{env}}'
source:
repoURL: https://github.com/myorg/gitops-deploy.git
targetRevision: main
path: 'apps/backend/overlays/{{env}}'
destination:
server:
Flux Kustomization
Flux uses native Kubernetes custom resources and supports multi-tenancy out of the box.
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: gitops-deploy
namespace: flux-system
spec:
interval: 1m
url: https://github.com/myorg/gitops-deploy.git
ref:
branch: main
secretRef:
name: gitops-deploy-auth
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: backend-prod
namespace: flux-system
spec:
interval: 5m
retryInterval: 2m
timeout: 5m
sourceRef:
kind: GitRepository
name: gitops-deploy
path: ./apps/backend/overlays/prod
prune: true
targetNamespace: backend
healthChecks:
- apiVersion: apps/v1
Progressive Delivery with Argo Rollouts
Argo Rollouts extends Kubernetes Deployments with canary releases, blue-green deployments, and automated analysis.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: backend
spec:
replicas: 10
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: backend
image: ghcr.io/myorg/backend:v1.2.3
ports:
- containerPort: 8080
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: 500m, memory: 512Mi }
strategy:
canary:
canaryService: backend-canary
stableService: backend-stable
trafficRouting:
istio:
virtualServices:
- name: backend-vsvc
routes: []
{ }
{ }
{ }
Reconciliation Loop
The reconciliation loop is the core mechanism that makes GitOps self-healing.
+----------------+
| Git Repo |
| (desired state)|
+-------+--------+
| poll / webhook
+-------+--------+
| GitOps Agent |
| (ArgoCD / Flux) |
+-------+--------+
| compare desired vs actual
+-------+--------+
| Diff Engine |----> No diff? Sleep, poll again.
+-------+--------+
| diff detected
+-------+--------+
| Apply Changes |
+-------+--------+
|
+-------+--------+
| Health Checks |----> Unhealthy? Retry or alert.
+-------+--------+
| healthy
+-------+--------+
| Mark Synced |
+----------------+
| Setting | ArgoCD | Flux |
|---|
| Poll interval | 3m default | spec.interval per resource |
| Webhook support | Yes (GitHub, GitLab) | Yes (Receiver CRD) |
| Self-heal | selfHeal: true | spec.force: true |
| Prune removed resources | prune: true | spec.prune: true |
| Retry on failure | retry.limit with backoff | retryInterval |
Environment Promotion Strategies
| Strategy | Mechanism | Pros | Cons |
|---|
| PR-based promotion | CI opens PR to update image tag | Full review, audit trail | Manual merge step |
| Automated promotion | Flux image automation or CI script | Fast, no manual gates | Needs strong automated tests |
| Git tag promotion | Tag commit for promotion | Clear versioning | Complex tag management |
| Branch promotion | Merge dev -> staging -> prod | Familiar Git workflow | Branch drift, merge conflicts |
PR-Based Promotion Pipeline
name: Promote
on:
workflow_dispatch:
inputs:
image_tag:
description: 'Image tag to promote'
required: true
target_env:
type: choice
options: [staging, prod]
jobs:
promote:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
steps:
- uses: actions/checkout@v4
with:
repository: myorg/gitops-deploy
token: ${{ secrets.GITOPS_PAT }}
- name: Update image tag
run: |
cd apps/backend/overlays/${{ inputs.target_env }}
kustomize edit set image \
backend=ghcr.io/myorg/backend:${{ inputs.image_tag }}
- uses: peter-evans/create-pull-request@v6
with:
title: "promote: backend ${{ inputs.image_tag }} to ${{ inputs.target_env }}"
Secret Management in GitOps
Secrets cannot be stored in plain text in Git. Multiple solutions exist with different trade-offs.
| Solution | Encryption | Where Decrypted | External Deps | Complexity |
|---|
| Sealed Secrets | Asymmetric (cluster key) | In-cluster controller | None | Low |
| SOPS + age/KMS | Symmetric or KMS | CI or GitOps agent | KMS optional | Medium |
| External Secrets Operator | None (fetched at runtime) | In-cluster controller | Vault, AWS SM | Medium |
| Vault CSI Provider | None (injected at runtime) | Sidecar / CSI driver | HashiCorp Vault | High |
Sealed Secrets
kubectl create secret generic db-credentials \
--from-literal=username=admin \
--from-literal=password=s3cret \
--dry-run=client -o yaml | \
kubeseal --format yaml \
--controller-name sealed-secrets \
--controller-namespace flux-system \
> apps/backend/overlays/prod/sealed-db-credentials.yaml
External Secrets Operator
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
namespace: backend
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: db-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: prod/backend/db
property: username
- secretKey: password
remoteRef:
key: prod/backend/db
property: password
SOPS with Flux Decryption
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: backend-prod
namespace: flux-system
spec:
interval: 5m
sourceRef:
kind: GitRepository
name: gitops-deploy
path: ./apps/backend/overlays/prod
prune: true
decryption:
provider: sops
secretRef:
name: sops-age-key
Anti-Patterns
| Anti-Pattern | Problem | Fix |
|---|
| Storing plain-text secrets in Git | Credentials exposed in repo history forever | Use Sealed Secrets, SOPS, or External Secrets Operator |
| Push-based deploys alongside GitOps | CI kubectl apply bypasses reconciliation | Remove all kubectl apply from CI; only update Git |
| Single repo for app code and manifests | Every code commit triggers deploy; noisy history | Separate application and deployment repositories |
| Branch-per-environment | Branches diverge, merge conflicts, unclear state | Use directory-per-environment with Kustomize overlays |
| Disabling self-heal / auto-prune | Manual changes accumulate, cluster drifts from Git | Enable selfHeal: true and prune: true |
| No health checks in sync policy | Sync marked complete before app is actually ready | Add health checks for Deployments, StatefulSets, Jobs |
Manual kubectl in GitOps clusters | Reverted on next reconciliation or causes drift | All changes through Git PRs; restrict kubectl to read-only |
| Deploying directly to prod | No validation in lower environments | Require promotion path: dev -> staging -> prod |
| No RBAC on GitOps tool | Any developer can sync any app to any namespace | Use ArgoCD AppProjects or Flux multi-tenancy |
Image tag latest in manifests | Non-deterministic deploys, no rollback possible | Use immutable tags (SHA, semver, build ID) |
| No notification on sync failure | Failed deployments go unnoticed for hours | Configure alerts to Slack, PagerDuty, or email |
| Reconciliation interval too long | Drift persists, delayed deployments | Set interval to 1-5 minutes; use webhooks for instant sync |
GitOps Readiness Checklist
