certificate-expiry-incident

name	certificate-expiry-incident
enabled	true
description	Use when performing certificate expiry incident — tLS/SSL certificate expiry incident response and prevention playbook. Covers emergency certificate renewal, impact assessment, interim mitigations, certificate chain validation, automated renewal configuration, monitoring setup, and organizational processes to prevent certificate-related outages.
required_connections	[{"prefix":"slack","label":"Slack (for incident coordination)"}]
config_fields	[{"key":"affected_domain","label":"Affected Domain","required":true,"placeholder":"e.g., api.example.com, *.example.com"},{"key":"certificate_provider","label":"Certificate Provider","required":false,"placeholder":"e.g., Let's Encrypt, DigiCert, ACM"},{"key":"expiry_status","label":"Expiry Status","required":true,"placeholder":"e.g., expired 2 hours ago, expires in 24 hours"}]
features	["INCIDENT"]

Certificate Expiry Incident Response

Domain: {{ affected_domain }} Provider: {{ certificate_provider }} Status: {{ expiry_status }}

Impact of Expired Certificates

• Browsers show security warnings, blocking user access
• API clients reject connections with TLS errors
• Mobile apps may hard-fail with certificate pinning errors
• Service-to-service communication breaks if mTLS is used
• Webhooks from third parties fail
• Search engine rankings can be affected

Phase 1 — Immediate Diagnosis (0-5 min)

Check Certificate Status

# Check certificate expiry for a domain
echo | openssl s_client -servername {{ affected_domain }} -connect {{ affected_domain }}:443 2>/dev/null | openssl x509 -noout -dates -subject -issuer

# Check certificate chain
echo | openssl s_client -servername {{ affected_domain }} -connect {{ affected_domain }}:443 -showcerts 2>/dev/null

# Check days until expiry
echo | openssl s_client -servername {{ affected_domain }} -connect {{ affected_domain }}:443 2>/dev/null | openssl x509 -noout -checkend 0
# Exit code 1 = expired, 0 = still valid

# Check from a specific file
openssl x509 -in /path/to/cert.pem -noout -dates

# Verify certificate matches private key
openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa -noout -modulus -in key.pem | openssl md5
# Both MD5 values must match

Identify All Affected Endpoints

• Which domains use this certificate? (check SANs)
• Is this a wildcard cert affecting multiple subdomains?
• Where is this certificate deployed? (load balancers, CDN, servers, containers)
• Are there any certificate-pinned clients?

Phase 2 — Emergency Renewal (5-30 min)

Option 1: Automated Renewal (Let's Encrypt / ACME)

# Certbot renewal
sudo certbot renew --cert-name {{ affected_domain }} --force-renewal

# Verify new certificate
sudo certbot certificates --cert-name {{ affected_domain }}

Option 2: Cloud Provider Managed Certificates

# AWS ACM — request new certificate
aws acm request-certificate \
  --domain-name {{ affected_domain }} \
  --validation-method DNS

# GCP — managed certificate auto-renews; check status
gcloud compute ssl-certificates describe CERT_NAME

# Azure — check App Service managed certificate
az webapp config ssl list --resource-group RG_NAME

Option 3: Manual Certificate Renewal

1. Generate new CSR:

openssl req -new -newkey rsa:2048 -nodes \
  -keyout {{ affected_domain }}.key \
  -out {{ affected_domain }}.csr \
  -subj "/CN={{ affected_domain }}"

2. Submit CSR to certificate provider ({{ certificate_provider }})
3. Complete domain validation (DNS or HTTP)
4. Download and install new certificate

Option 4: Emergency Self-Signed (last resort, internal only)

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout emergency.key -out emergency.crt \
  -days 30 -subj "/CN={{ affected_domain }}"

WARNING: Self-signed certificates will cause browser warnings and API client failures. Use only for internal services as a temporary measure.

Phase 3 — Deploy New Certificate

Deployment Checklist

• New certificate validated (correct domain, valid chain, key matches)
• Deploy to load balancer(s)
• Deploy to CDN edge (if applicable)
• Deploy to application servers (if terminating TLS)
• Update Kubernetes secrets (if applicable)
• Restart/reload web servers (nginx, Apache, etc.)
• Flush any TLS session caches

Verification After Deployment

# Verify new certificate is being served
echo | openssl s_client -servername {{ affected_domain }} -connect {{ affected_domain }}:443 2>/dev/null | openssl x509 -noout -dates -subject

# Test HTTPS connectivity
curl -vI https://{{ affected_domain }} 2>&1 | grep -E "expire|subject|issuer|SSL"

# Check certificate chain is complete
echo | openssl s_client -servername {{ affected_domain }} -connect {{ affected_domain }}:443 2>/dev/null | grep -E "Verify|depth"

Verification Checklist

• Certificate shows new expiry date
• Full certificate chain is valid
• No mixed content warnings
• API clients connecting successfully
• Monitoring tools reporting healthy
• SSL Labs test passing (https://www.ssllabs.com/ssltest/)

Phase 4 — Prevention

Automated Renewal Setup

• Configure ACME/certbot auto-renewal with cron or systemd timer
• Use cloud-managed certificates where possible (ACM, GCP managed)
• Test renewal process in staging before relying on it

Monitoring and Alerting

• Set up certificate expiry monitoring (alert at 30, 14, 7, 3, 1 days)
• Monitor certificate validity from external vantage points
• Add certificate expiry to team dashboard
• Configure alerts for renewal failures

Organizational Process

• Maintain certificate inventory (domain, provider, expiry, owner, location)
• Assign certificate ownership to specific teams
• Include certificate review in quarterly security audits
• Document renewal procedures in runbook
• Add certificate checks to production readiness review

Counter-Rationalizations

Shortcut	Counter	Why
"We can skip some steps for this case"	Adapt the workflow steps, don't skip them	Skipped steps are where incidents and oversights originate
"The user seems to already know what to do"	Complete all workflow phases with the user	The workflow catches blind spots that experience alone misses
"This is a minor case, full process is overkill"	Scale the process down, don't turn it off	Minor cases become major when unstructured; the process scales, not disappears
"I'll fill in the details later"	Complete each section before moving on	Deferred details are forgotten; real-time capture is more accurate
"The template output isn't necessary"	Always produce the structured output format	Structured output enables comparison, audit trails, and handoff to other teams