This skill covers deploying HashiCorp Vault for centralized secrets management across cloud environments, including dynamic secret generation for databases and cloud providers, transit encryption, PKI certificate management, and Kubernetes integration. It addresses eliminating hardcoded credentials from application code and CI/CD pipelines by implementing short-lived, automatically rotated secrets.
This skill covers deploying HashiCorp Vault for centralized secrets management across cloud environments, including dynamic secret generation for databases and cloud providers, transit encryption, PKI certificate management, and Kubernetes integration. It addresses eliminating hardcoded credentials from application code and CI/CD pipelines by implementing short-lived, automatically rotated secrets.
When applications store database passwords, API keys, or certificates in environment variables or config files
When migrating from static long-lived credentials to dynamic short-lived secrets
When Kubernetes workloads need secure access to database credentials or cloud provider APIs
When compliance requirements mandate centralized credential management with audit logging
When CI/CD pipelines contain hardcoded secrets that represent supply chain risk
Do not use for AWS-only environments where AWS Secrets Manager suffices without multi-cloud requirements, for application-level encryption logic (though Vault Transit can help), or for identity federation (see managing-cloud-identity-with-okta).
Common Misconfigurations & Verification
Root token not revoked after init: the initial root token from vault operator init stays valid forever if left around. Confirm vault token lookup <root-token> errors out; revoke with vault token revoke -self and rely on OIDC/AppRole for ongoing access.
Root DB credentials never rotated: if you skip vault write -force database/rotate-root/<db>, the bootstrap password in connection_url still works outside Vault. Verify by attempting a direct login with the original password - it must fail.
secret_id_num_uses=0 / secret_id_ttl=0: unlimited, non-expiring AppRole SecretIDs defeat short-lived auth. Audit with vault read auth/approle/role/<name> and flag any 0 values.
Audit device disabled or single sink: if vault audit list is empty, secret access is unlogged. Enable at least one file plus one syslog/socket device so a single sink outage doesn't silence the trail.
Overbroad policies: a policy missing the path "sys/*" { capabilities = ["deny"] } rule or granting sudo leaks admin paths. Run vault policy read <name>, then vault token create -policy=<name> and probe the paths that should be denied.
TTLs longer than the workload: dynamic DB creds with default_ttl=24h outlive most jobs. Keep TTLs near actual usage and confirm a test lease auto-revokes with vault lease lookup/.
vault lease revoke
Kubernetes auth pinned to a long-lived reviewer JWT: prefer the short-lived projected SA token; verify auth/kubernetes/config does not hardcode a non-expiring token_reviewer_jwt.
Don't declare it secured until the root token is revoked, every root/static credential Vault now manages is rotated, audit logging survives a sink failure, and a test lease is shown to auto-revoke at TTL.
Prerequisites
HashiCorp Vault server deployed in HA mode (Consul or Raft storage backend)
TLS certificates for Vault listener endpoints
Vault Enterprise license for namespaces, Sentinel policies, and replication (optional)
Kubernetes cluster with Vault Agent Injector or CSI provider for workload integration
Workflow
Step 1: Deploy Vault in High Availability Mode
Deploy Vault using Integrated Storage (Raft) for HA without external dependencies. Configure TLS, audit logging, and auto-unseal using a cloud KMS.
Enable authentication backends for human operators, applications, and CI/CD pipelines. Use AppRole for machine authentication and OIDC for human access.
Configure database secret engines to generate short-lived credentials on demand. Each credential set has a TTL and is automatically revoked when it expires.
# Enable database secrets engine for PostgreSQL
vault secrets enable database
vault write database/config/production-db \
plugin_name=postgresql-database-plugin \
allowed_roles="readonly,readwrite" \
connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/production?sslmode=require" \
username="vault_admin" \
password="initial-password"# Rotate the root credentials so Vault manages them exclusively
vault write -force database/rotate-root/production-db
# Create a readonly role with 1-hour TTL
vault write database/roles/readonly \
db_name=production-db \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
revocation_statements="REVOKE ALL ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"# Enable AWS secrets engine for dynamic IAM credentials
vault secrets enable aws
vault write aws/config/root \
access_key=AKIAEXAMPLE \
secret_key=secretkey \
region=us-east-1
vault write aws/roles/deploy-role \
credential_type=iam_user \
policy_document=@deploy-policy.json \
default_sts_ttl=3600
Step 4: Integrate with Kubernetes Workloads
Use the Vault Agent Injector or CSI Provider to deliver secrets to pods without application code changes. Secrets are rendered as files in a shared volume.
Use the Transit secrets engine for application-level encryption without managing keys in application code. Deploy the PKI engine for automatic TLS certificate management.
# Enable Transit engine for encryption as a service
vault secrets enable transit
vault write -f transit/keys/payment-data type=aes256-gcm96
# Encrypt sensitive data
vault write transit/encrypt/payment-data \
plaintext=$(echo"card-number-4111-1111-1111-1111" | base64)
# Enable PKI for internal certificate management
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki
# Generate root CA
vault write pki/root/generate/internal \
common_name="Internal Root CA" \
ttl=87600h
# Configure intermediate CA for issuing certificates
vault secrets enable -path=pki_int pki
vault write pki_int/intermediate/generate/internal \
common_name="Internal Intermediate CA" \
ttl=43800h
# Create a role for issuing certificates
vault write pki_int/roles/internal-services \
allowed_domains="internal.company.com" \
allow_subdomains=true \
max_ttl=720h
Step 6: Establish Policies and Audit Trail
Define fine-grained ACL policies following least privilege. Enable comprehensive audit logging for all secret access and administrative operations.
Vault Agent Injector: Kubernetes mutating webhook that automatically injects Vault secrets into pod volumes via sidecar containers
Vault CSI Provider: Kubernetes CSI driver that mounts Vault secrets directly into pod volumes without sidecar containers
consul-template: Template rendering daemon that watches Vault secrets and re-renders configuration files when secrets change
Vault Radar: Secret scanning tool that detects hardcoded credentials in source code, CI/CD pipelines, and cloud configurations
Common Scenarios
Scenario: Eliminating Hardcoded Database Credentials from CI/CD Pipeline
Context: A DevOps team stores PostgreSQL credentials in GitHub Actions secrets and Jenkins credential stores. The same credentials are shared across staging and production environments with no rotation for 18 months.
Approach:
Deploy Vault with AppRole auth enabled for CI/CD systems
Configure the database secrets engine with separate roles for staging (readwrite, 2h TTL) and production (readonly, 1h TTL)
Create separate Vault policies for each pipeline stage restricting access to the appropriate database role
Update GitHub Actions workflows to authenticate via AppRole and request dynamic credentials at the start of each job
Rotate the static PostgreSQL credentials and hand root access to Vault exclusively
Enable audit logging to track every credential request with pipeline job metadata
Pitfalls: Failing to rotate the original static credentials after Vault migration leaves the old credentials valid. Setting TTLs too short causes credential expiry mid-deployment for long-running jobs.
Output Format
Vault Secrets Management Audit Report
=======================================
Vault Cluster: vault.internal.company.com
Version: 1.18.1 Enterprise
HA Mode: Raft (3 nodes)
Seal Type: AWS KMS Auto-Unseal
Report Date: 2025-02-23
SECRET ENGINES:
database/ PostgreSQL dynamic creds Leases Active: 47
aws/ Dynamic IAM credentials Leases Active: 12
transit/ Encryption as a service Keys: 8
pki/ Root CA Certs Issued: 0
pki_int/ Intermediate CA Certs Issued: 234
secret/ KV v2 static secrets Versions: 1,892
AUTH METHODS:
oidc/ Okta SSO for humans Active Tokens: 23
approle/ CI/CD pipelines Active Tokens: 156
kubernetes/ Pod-based auth Active Tokens: 89
AUDIT FINDINGS:
[WARN] 3 AppRole secret_id_num_uses set to 0 (unlimited)
[WARN] 12 KV secrets not accessed in 90+ days (potential orphans)
[PASS] All dynamic secret TTLs under 24 hours
[PASS] Audit logging enabled on all nodes
[PASS] Root token revoked after initial setup
CREDENTIAL HYGIENE:
Static Secrets (KV): 234
Dynamic Secrets Active: 59
Average Lease TTL: 2.3 hours
Secrets Rotated This Month: 12,456
