Deep secret-management operational intuition — tradeoffs across sops+age, KMS, Vault, Sealed Secrets, External Secrets Operator, k8s Secrets; rotation, threat boundaries, leak detection, workflow specifics. Load when choosing approaches, designing rotation, configuring sops/Vault/ESO/Sealed Secrets, etcd encryption, or diagnosing leak/propagation/bootstrap problems. Skip for "store this password" or trivial `kubectl create secret`. Triggers on: "sops", "age encryption", "AppRole", "sealed secrets", "ExternalSecret", "kubeseal", "etcd encryption at rest", "dynamic credentials", "gitleaks", "auto-unseal".
Secrets Management Operational Guide
Concise pointers for choosing and operating secret-management systems.
Assumes you know what an env var, a k8s Secret, and an IAM role are. This skill covers the operational layer — the parts models tend to gloss over: which tool fits which threat model, the workflow specifics of sops / Vault / ESO / Sealed Secrets, rotation mechanics, leak-detection tooling, and boot-strap-trust pitfalls.
When to use
Load when the question is about:
Picking among sops+age / Vault / Sealed Secrets / ESO / SaaS for a specific environment
sops workflow (.sops.yaml, encrypted_regex, updatekeys, age recipients)
Operates a stateful HA cluster; seal/unseal ceremony; full-time operator
AWS SM / GCP SM / Azure KV
Already in that cloud, want minimal ops
Cloud lock-in; per-secret cost; cross-region replication is extra
Sealed Secrets
Pure-k8s GitOps; secrets must live in Git encrypted
Controller-only decryption; lose master key = lose all secrets; namespace+name binding by default
External Secrets Operator
Already running a secret store; want k8s-native sync
Bootstrap-trust problem (auth to backend); k8s Secret object still on cluster
Doppler / Infisical / 1Password Automation
Small team, want polished UX, no infra
Vendor outage = your secrets are gone; data residency questions
Rule of thumb: pick the simplest tool that satisfies the threat model. sops+age beats Vault for a 3-person team; Vault beats sops the moment you need dynamic database creds or PKI.
sops
What it encrypts: VALUES of structured files (YAML/JSON/INI/ENV/dotenv); KEYS stay plaintext. Preserves diff readability — reviewing PRs against encrypted files still works.
.sops.yaml creation_rules: list evaluated top-down, first match wins. Each rule has path_regex, recipient set (age: / kms: / pgp: / hc_vault_transit_uri:), and optional encrypted_regex / unencrypted_regex (mutually exclusive).
Selective encryption: encrypted_regex 'password|token|secret|key' encrypts only matching keys; the rest stays plaintext. Useful for k8s Secret manifests where metadata should remain navigable.
Commands: sops -e file.yaml encrypt; sops file.yaml open in $EDITOR (auto decrypt/re-encrypt); sops -d file.yaml decrypt to stdout; sops -i -e file.yaml in-place encrypt.
MAC: SOPS computes a MAC over encrypted values + key paths as AAD → tamper detection. Modifying the YAML structure of an encrypted file breaks the MAC.
Per-value AES-256-GCM with unique IV: prevents pattern analysis across the file.
Key rotation: edit .sops.yaml recipient list, then sops updatekeys file.yaml (re-encrypts the data key to new recipients without changing the data key itself). For a full data-key rotation use sops --rotate -i file.yaml or sops rotate.
Shamir threshold: key_groups + shamir_threshold: N → require N of M groups to decrypt. Use when no single team should be able to read prod alone.
Key file lookup order (age): SOPS_AGE_KEY_FILE env var → SOPS_AGE_KEY (raw key) → SOPS_AGE_KEY_CMD (command output) → ~/.config/sops/age/keys.txt.
age
Pure file encryption, X25519. Intentionally minimal: no signing, no negotiation, no algorithm agility.
age-keygen -o key.txt writes private key to file; line 2 is # public key: age1... (the recipient).
age -e -r age1abc... -o out.age in.txt encrypt to one recipient.
age -d -i key.txt out.age decrypt with identity file.
Multiple recipients: -r is repeatable; resulting file decrypts with ANY one identity. Pattern: encrypt to every developer's age pubkey + a CI/CD key.
SSH keys as recipients: age -R ~/.ssh/authorized_keys -o out.age in.txt — supports ssh-ed25519 and ssh-rsa. Decrypt with age -d -i ~/.ssh/id_ed25519.
Recipient files: -R recipients.txt (one per line, comments OK).
No revocation: removing a recipient from .sops.yaml and running sops updatekeys does NOT erase historical Git-committed encrypted files — the old recipient can still decrypt those revisions. Treat removed recipient = compromised data.
HashiCorp Vault
Engines
KV v1: no versioning. vault kv put secret/foo k=v, vault kv get secret/foo. Overwrites lose history.
KV v2: versioned, soft-delete, configurable max_versions. vault kv get -version=3 secret/foo. Old versions retained until destroy.
Database: dynamic credentials with TTL. Configure connection + role (creation_statements SQL with {{username}}/{{password}} placeholders), then vault read database/creds/<role> returns ephemeral user with lease_id + lease_duration. Default TTL 1h, max 24h. Supports Postgres, MySQL/MariaDB, Mongo, MSSQL, Oracle, Cassandra, Redis, Snowflake, Elasticsearch, +20.
PKI: issue short-lived X.509 certs with vault write pki/issue/<role> common_name=.... CRL/OCSP managed by Vault.
Transit: encryption-as-a-service. App sends plaintext, gets ciphertext back; key never leaves Vault. Use for application-level encryption when you don't want to ship key material.
SSH: signed-cert auth — Vault signs an SSH cert using its CA, host trusts the CA, no per-user public-key dance.
Auth methods
Token: native, including the root token (printed at init/dev). Never use root token for routine ops.
Kubernetes: pod presents its SA JWT; Vault validates against the k8s API (tokenreviews). Tied to namespace + SA via role. No long-lived secret in cluster.
AWS IAM: instance profile or IAM principal signs sts:GetCallerIdentity; Vault verifies signature.
OIDC/JWT: federate from any OIDC IdP (GitHub, GitLab, Auth0, Okta, OAuth2 proxies).
Shamir (default): unseal key split into N shares, M required to reconstruct (configurable at vault operator init, e.g. -key-shares=5 -key-threshold=3). Each operator runs vault operator unseal with one share.
Auto-unseal: delegate to AWS KMS, GCP KMS, Azure KV, OCI KMS, AliCloud KMS, HSM (PKCS#11), or Transit on a separate Vault. With auto-unseal, Vault generates recovery keys (not unseal keys) — they cannot decrypt the root key, only authorize emergency operations.
Critical: deleting the seal KMS key = unrecoverable cluster, even from backup.
Operational essentials
-dev mode: in-memory, root token printed to stdout, no TLS. NEVER production. Resets on restart.
Audit devices: file / syslog / socket. Required for compliance; multiple can run; if all audit devices fail, Vault BLOCKS requests (intentional). vault audit enable file file_path=/var/log/vault_audit.log.
Response wrapping (cubbyhole): vault wrap returns a one-time token in a per-token cubbyhole. Used for hand-off (e.g., delivering secret_id to a worker without trusting the courier).
Lease management: vault list sys/leases/lookup/..., vault lease renew <id>, vault lease revoke <id>. Dynamic creds revoke on lease expiry — design apps to fetch fresh creds + retry on permission denied, not to cache forever.
Sealed Secrets (Bitnami)
Controller-side asymmetric: controller in cluster generates RSA keypair on first start; pubkey distributed via kubeseal --fetch-cert. Only the controller can decrypt.
kubeseal -f secret.yaml -w sealed.yaml produces a SealedSecret CR safe to commit. Apply: kubectl apply -f sealed.yaml → controller decrypts → creates real Secret.
cluster-wide: any namespace, any name. Loosest; only when you must.
Rotation: controller generates new key every 30 days (--key-renew-period). Old keys retained to decrypt existing SealedSecrets. To re-encrypt with the latest key without exposing plaintext: kubeseal --re-encrypt < old.yaml > new.yaml.
Master-key backup (NON-NEGOTIABLE): kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > master.key. Store offline. Without this, a wiped cluster cannot decrypt any committed SealedSecret.
Limitation: can only encrypt a Secret shape — for non-Secret resources you need ESO or sops.
refreshInterval: re-fetches and updates the k8s Secret. 0 = create once, never refresh.
dataFrom: bulk import; takes whole JSON object as keys, or supports extract / find patterns.
creationPolicy: Owner (default — k8s Secret is owned by the CR, deletes cascade) / Merge (update existing, no owner) / Orphan (create no owner refs) / None (injector use only).
deletionPolicy: Retain (default) / Delete (drop k8s Secret when remote keys gone) / Merge (drop only the keys, keep Secret).
Auth bootstrap problem: SecretStore needs creds to talk to the backend. Solutions in order of preference:
Workload identity / IRSA (EKS) / Workload Identity Federation (GKE) — pod assumes IAM role via OIDC; no secret in cluster.
Kubernetes auth method (Vault) — pod's SA JWT validated by Vault; no secret in cluster.
Static token in a k8s Secret (chicken-and-egg; only acceptable for boot-strap).
Generators (push-based): Password, ECRAuthorizationToken, GCRAccessToken, ACRAccessToken, Webhook, ClusterGenerator. Password produces a generated password into a Secret; ECRAuthorizationToken produces a 12-hour Docker pull cred without static creds in cluster.
Kubernetes native Secrets
base64 is encoding, NOT encryption. By default Secrets land in etcd as plaintext (base64 in YAML, plaintext on disk).
Encryption at rest: --encryption-provider-config=/etc/k8s/enc.yaml on kube-apiserver. Provider chain — first encrypts, all decrypt:
Providers: aescbc (AES-CBC + PKCS#7 + HMAC), secretbox (XChaCha20-Poly1305), kms v1/v2, identity (none). KMS v1 deprecated since 1.28, disabled by default 1.29 — use kms v2 (automatic seed rotation, KDF-derived per-write DEKs, no manual cachesize).
Provider order: first listed encrypts new writes; reads try each in order. Add new provider FIRST, then re-encrypt all secrets (kubectl get secret -A -o json | kubectl replace -f -), THEN remove old.
immutable: true (1.21+): Secret cannot be updated or have data changed; reduces apiserver/etcd watch load. Useful for rarely-changed long-lived secrets.
Update propagation:
Mount type
Updates?
Latency
Volume mount
yes
~kubelet sync, default 60s
Volume mount + subPath
NO
requires pod restart
Env var (valueFrom.secretKeyRef)
NO
baked at pod start
/proc/PID/environ leak: any process with same UID can read another's env vars. Volume mounts (mode: 0400) are safer than env vars for sensitive material.
Rotation patterns
Static rotation (passwords, API keys): rotate in store + redeploy/restart consumers. Outage window = consumer restart time. Decouple by keeping N+1 versions live (consumers read latest; old continues to work briefly).
Dynamic credentials (Vault DB engine, AWS STS): consumers fetch fresh creds with TTL. Rotation is automatic when lease expires. Apps must handle "creds expired mid-flight" — typically retry-on-401 against fetch-creds endpoint.
Versioned secrets (KV v2, AWS SM versioned): old version retained as AWSPREVIOUS / version=N-1; consumers can pin or follow latest. Rotation = create new version, then update consumer pointer.
IAM federation (no rotation): OIDC trust between IdP and AWS/GCP/Azure → short-lived STS tokens (15min–12h). Best pattern when both parties speak OIDC. No secret to rotate because no secret exists.
Database root rotation (Vault rotate-root): Vault rotates the DB root password to a value only Vault knows. Subsequent DB engine ops still work; humans permanently lose root unless they break-glass.
Trust boundaries / threat model
At rest: disk encryption (FDE) is necessary but not sufficient. KMS-encrypted secret value defends against backup theft and operator browsing etcd.
In transit: TLS to the secret store. Verify cert chain — a Vault client trusting InsecureSkipVerify=true defeats the design.
In use: secret in process memory. For high-stakes (banking, PII), TEEs (AWS Nitro Enclaves, Intel SGX) keep keys out of host kernel reach.
Supply chain: who can read the store? CI runners with broad pull access are a wide blast radius. Apply per-job least-privilege (per-environment AppRole / per-job OIDC subject claim).
trufflehog: similar surface, plus active credential verification — calls the suspected provider to check if the leaked key actually works. Lower false positive rate at cost of network calls.
GitHub secret scanning: built-in for public repos, free; partner program for private repos pushes alerts to providers (e.g., AWS auto-quarantines leaked keys).
Env-var secrets readable via /proc/PID/environ (same UID). Prefer file mounts with 0400.
docker build --build-arg PASSWORD=...: build args persist in image layers / docker history. Use BuildKit --mount=type=secret,id=foo instead — secret only available during the RUN instruction at /run/secrets/foo, not in image. CLI: docker build --secret id=foo,src=./secret.txt.
Logging exception messages with secret values — sanitize at the log boundary, not the call site.
kubectl edit secret shows base64: a quick base64 -d reveals plaintext in terminal scrollback. Clear scrollback / use kubectl create secret --dry-run=client -o yaml | kubeseal instead.
Sealed Secrets controller key not backed up → cluster rebuild = unrecoverable secrets. Back up kube-system/sealed-secrets-key* immediately on day 1.
ESO SecretStore static-token auth in cluster: chicken-and-egg, defeats the point. Use IRSA / Workload Identity / Vault k8s auth instead.
sops + age committed without .sops.yaml: new contributors can decrypt (their key is in the file) but cannot encrypt new files correctly because they don't know the recipient list. Always commit .sops.yaml.
Rotating an age recipient list does not retroactively re-key history: removed recipients still decrypt old Git revisions. Treat any removed recipient as having permanent access to anything they could read at removal time.
k8s Secret updates don't propagate to subPath mounts or env vars — surprise stale creds after rotation. Use plain volume mounts when rotation matters.
