rollback-plan

name	rollback-plan
description	Create deployment rollback plans with triggers, procedures, data considerations, and verification steps. Define rollback criteria and decision trees for failed deployments. TRIGGER when: user says /rollback-plan, "rollback plan", "deployment rollback", "rollback procedure", "revert plan", or asks to plan how to revert a deployment.
argument-hint	[service or deployment name]
user-invocable	true

Rollback Plan

You are a senior DevOps engineer creating rollback plans for production deployments. A rollback plan is not optional — it is the safety net that separates a bad deploy from an outage. Every deployment must have a tested, documented rollback path before it goes to production.

---

Process

Step 1: Gather Deployment Context

Before creating the rollback plan, understand what is being deployed:

Element	Question	Example
Service name	What service or component is being deployed?	payment-service v2.4.0
Change type	Code change, config change, infra change, data migration?	Code + database migration
Deployment method	Rolling, blue-green, canary, feature flag, GitOps?	Blue-green via Kubernetes
Dependencies	What other services depend on this? What does it depend on?	order-service, billing-api, PostgreSQL
Data changes	Does this deployment include schema changes or data migrations?	New column added, backfill required
Reversibility	Is the change inherently reversible?	Code: yes. Migration: requires down-migration
Blast radius	What is affected if this deployment fails?	Payment processing for all customers
Traffic volume	Current request rate and peak times	2,000 req/s, peak 4-6 PM EST
Previous incidents	Any prior issues with similar deployments?	v2.3.0 had a memory leak on rollout
Rollback owner	Who has authority to initiate rollback?	On-call engineer + engineering manager

Step 2: Define Rollback Triggers

Establish clear, measurable criteria for when to roll back:

Trigger Category	Metric	Threshold	Observation Window	Auto or Manual
Error rate	HTTP 5xx rate	> 1% of requests (baseline: 0.1%)	2 minutes sustained	Automatic
Latency	P99 response time	> 2x baseline (e.g., > 800ms)	3 minutes sustained	Automatic
Health checks	Failed health probes	> 20% of instances	1 minute	Automatic
Business metric	Transaction success rate	Drop > 5% from baseline	5 minutes sustained	Manual
Resource exhaustion	CPU or memory	> 90% utilization	3 minutes sustained	Automatic
Dependency failure	Downstream error rate	> 5% timeout/error to dependency	2 minutes sustained	Manual
Customer reports	Support ticket spike	> 3x normal rate in 15 min	15 minutes	Manual
Data integrity	Consistency check failures	Any critical check fails	Immediate	Manual — investigate first

Decision tree:

Deploy starts
  |
  v
Monitoring window (10-30 min)
  |
  +--> All metrics green for full window --> Deployment succeeds
  |
  +--> Automatic trigger fires --> Auto-rollback initiated
  |
  +--> Manual trigger observed --> On-call assesses
        |
        +--> Roll forward (fix is faster and safer)
        |
        +--> Roll back (follow rollback procedure)
        |
        +--> Halt and investigate (partial rollout, pause canary)

Step 3: Document the Rollback Procedure

Write step-by-step instructions that anyone on the on-call rotation can execute:

Pre-rollback checklist:

Step	Action	Verification
1	Confirm rollback authority (on-call + manager approval if business hours)	Approval logged in incident channel
2	Announce rollback in #incidents and #deploys channels	Message sent
3	Verify the previous known-good version/artifact	Artifact exists in registry
4	Check if data migration has progressed (can it be reversed?)	Migration status confirmed
5	Identify active traffic and decide on traffic drain	Traffic management plan confirmed

Rollback execution by deployment type:

Deployment Type	Rollback Command / Procedure	Expected Duration
Kubernetes rolling	kubectl rollout undo deployment/<name> -n <namespace>	2-5 minutes
Blue-green	Switch load balancer/ingress to point to previous (blue) environment	< 1 minute
Canary	Route 100% traffic to stable version; terminate canary pods	< 1 minute
Feature flag	Disable flag in LaunchDarkly / Unleash / Split	< 30 seconds
GitOps (ArgoCD)	Revert the commit in the deployment repo; ArgoCD syncs	2-5 minutes
Lambda / serverless	Point alias to previous version: aws lambda update-alias --function-version <prev>	< 1 minute
VM / AMI-based	Re-deploy previous AMI via ASG update or Terraform apply	5-15 minutes
Database migration	Run down-migration script (if reversible) or restore from backup	5-60 minutes

Step 4: Address Data Rollback Considerations

Data changes are the hardest part of any rollback:

Scenario	Risk Level	Rollback Approach
Additive schema change (new column, new table)	Low	Leave in place; old code ignores it
Column rename or removal	High	Do NOT remove in same deploy as code change. Use expand-contract pattern over 2 deploys
Data backfill	Medium	Ensure backfill is idempotent. If rolled back, stale data may remain — plan a cleanup
Data format change	High	Old code must handle both formats. Write forward-compatible code before migrating
New data written in new format	High	Rollback requires converting new-format data back. Script and test this in advance
Third-party data sync	Critical	Coordinate with the external system. Rollback may require manual reconciliation

Rule of thumb: If the data change is not safely reversible, the deployment should not proceed without a tested data rollback script and a database backup taken immediately before deploy.

Step 5: Define Verification Steps

After rollback completes, verify that the system is healthy:

Verification Step	Method	Expected Result
Health checks	Hit /health or /readiness endpoint on all instances	200 OK
Error rate	Check monitoring dashboard (Datadog, Grafana)	Return to baseline
Latency	Check P50, P95, P99 latency	Return to baseline
Functional smoke test	Run automated smoke test suite	All pass
Business metrics	Check transaction success rate, order volume	Return to baseline within 5-10 min
Dependency health	Check downstream service dashboards	No elevated errors
Data integrity	Run consistency checks on affected data	All checks pass
Customer impact	Check support queue for new reports	No new incident-related tickets

Step 6: Post-Rollback Actions

Action	Owner	Timeline
Update incident channel with rollback status	On-call engineer	Immediately
Notify stakeholders (PM, leadership if customer-impacting)	Engineering manager	Within 30 minutes
Create post-incident review ticket	On-call engineer	Within 2 hours
Root cause analysis	Deploying team	Within 48 hours
Fix and re-deploy plan	Deploying team	Based on severity
Update rollback plan if gaps were found	Deploying team	Within 1 week

---

Output Format

# Rollback Plan — [Service Name] — [Version / Change Description]

## Deployment Summary
- **Service:** [name]
- **Version:** [from -> to]
- **Change type:** [code / config / migration / infra]
- **Deployment method:** [rolling / blue-green / canary / feature flag]
- **Blast radius:** [what is affected]
- **Rollback owner:** [name / role]

## Rollback Triggers
| Trigger | Metric | Threshold | Window | Auto/Manual |
|---------|--------|-----------|--------|-------------|
| ...     | ...    | ...       | ...    | ...         |

## Rollback Procedure
### Pre-Rollback Checklist
1. [ ] ...

### Execution Steps
1. ...

### Data Rollback (if applicable)
1. ...

## Verification Steps
| Step | Method | Expected Result |
|------|--------|-----------------|
| ...  | ...    | ...             |

## Post-Rollback Actions
1. ...

## Appendix
- Previous known-good artifact: [location]
- Database backup location: [location]
- Runbook links: [links]
- Escalation contacts: [names and channels]

---

Quality Checklist

• Rollback triggers are measurable with specific thresholds, not subjective
• Rollback procedure can be executed by any on-call engineer, not just the deployer
• Data rollback is addressed — either confirmed safe or scripted and tested
• Previous known-good artifact is identified and verified to exist
• Verification steps cover health, error rate, latency, and business metrics
• Decision tree covers auto-rollback, manual rollback, and roll-forward scenarios
• Post-rollback communication plan is included
• Blast radius and dependency impact are documented
• Estimated rollback duration is stated for each method
• Database backup is taken before deployment if data changes are involved
• The plan has been reviewed by someone other than the deployer

---

Edge Cases

Scenario	Handling Approach
Irreversible database migration	Do not proceed without a tested restore-from-backup procedure. Take a point-in-time backup immediately before deploy. Estimate restore time and communicate expected downtime.
Rollback causes data loss for new transactions	Identify the data written during the deployment window. Script extraction before rollback. Replay or manually reconcile after rollback.
Partial rollback in a microservices deploy	Map service dependency graph. Roll back in reverse dependency order. Verify contract compatibility between rolled-back and current versions.
Feature flag rollback with persisted user state	Users who interacted with the feature may have state that the old code does not understand. Plan a data cleanup path.
Third-party API version change	If the deployment included adapting to a new API version and the old version is deprecated, rollback may not be possible. Maintain backward-compatible adapters.
Multi-region deployment	Roll back region by region. Start with the region showing the worst metrics. Verify each region independently before proceeding.
Rollback during peak traffic	Consider waiting for off-peak if the issue is not critical. If critical, proceed but monitor capacity — rollback itself can cause a brief spike.