webhook-secret-rotation

name	webhook-secret-rotation
description	Rotate webhooks, API keys, and shared secrets fleet-wide without breaking consumers. The pattern: list all consumers, add overlap (old key + new key valid for 7 days), migrate consumers to new key, revoke old key. Use when an integration is compromised or as a periodic security hygiene step.

Webhook Secret Rotation

When a shared secret (webhook, API key, OAuth client secret) needs to be rotated, the goal is zero downtime — the system stays up the entire time. The pattern is to add the new secret alongside the old, migrate consumers, then revoke the old.

The 4-Phase Pattern

Phase 1: Discovery (Day 0)

Before rotating, find every consumer of the secret:

# For GitHub webhooks, look in the target repo's workflow files
gh api repos/camster91/<repo>/contents/.github/workflows | jq -r '.[] | .name' | while read f; do
    gh api repos/camster91/<repo>/contents/.github/workflows/$f 2>/dev/null | \
        jq -r '.content' | base64 -d 2>/dev/null | \
        grep -E '(secrets\.[A-Z_]+|api[_-]?key|token|webhook)' | head -5
done

# For OAuth/API keys, grep the codebase
grep -rE "(api[_-]?key|client[_-]?secret|webhook[_-]?secret|token).*=" --include="*.env*" --include="*.yml" --include="*.json" .

Build a list:

• Which repos use this secret
• Where (env var, file, config)
• How often it's called (every request? once on deploy?)

Phase 2: Add the new secret (Day 0)

Add the new secret alongside the old. Update the secret store to accept BOTH for 7 days:

# Example: GitHub webhook secret
gh api repos/camster91/<owner>/<repo>/hooks/<hook_id> \
    -X PATCH \
    -d '{"secret": "new_secret_here"}'
# Now the webhook accepts BOTH old and new (if your receiver does the check)

For API keys, the pattern is to add API_KEY_NEW and API_KEY_OLD env vars, and have the application try the new first, then fall back to the old.

Phase 3: Migrate consumers (Days 1-6)

Roll out the new secret to each consumer. For webhooks, this is automatic once the secret store accepts both. For API keys, you need to:

• Push a code change to each consumer that switches to the new env var name
• Deploy to production
• Verify the new env var is being used

Phase 4: Revoke the old (Day 7+)

Once all consumers are confirmed migrated, revoke the old secret:

# For GitHub webhooks, just update to only the new one
gh api repos/camster91/<owner>/<repo>/hooks/<hook_id> \
    -X PATCH \
    -d '{"secret": "new_secret_here"}'
# Now the old secret is no longer accepted

# For OAuth, rotate the client secret in the provider's dashboard
# Then update all consumers to use the new one

Common Mistakes

1. Don't revoke the old secret immediately — consumers that haven't migrated will break
2. Don't skip the overlap window — even 24 hours is better than zero
3. Don't store old and new secrets with the same name — use OLD_WEBHOOK_SECRET and NEW_WEBHOOK_SECRET to track which is which
4. Don't forget to update docs/runbooks that reference the secret by name

For Specific Secret Types

GitHub Webhooks

• Update via gh api /repos/{owner}/{repo}/hooks/{id} PATCH
• Receiver must accept BOTH old and new for the overlap window
• Use HMAC-SHA256 signature comparison

OAuth Client Secrets

• Generate new in provider's dashboard
• Update redirect URI if needed
• Consumers must update env vars AND deploy
• Old secret can be revoked once all consumers confirm migration

Database Passwords

• Add new user with same permissions
• Update connection strings
• Test with new user in non-prod first
• Drop old user after migration

Verification

After rotation:

1. All consumers confirmed using new secret
2. Old secret no longer accepted (test by trying to use it)
3. Audit logs show no failed auth attempts in the overlap window (indicates smooth rotation)
4. Docs updated to reference new secret

Related Skills & Chains

• secret-scanner — Use this first to find where the secret is currently used
• fleet-ci-audit — Confirms the rotation didn't break any CI workflows