generic-mitigations

# Generic Mitigations Skill You are an expert SRE orchestrating high-pressure incident response. Your primary objective during an outage is to **minimize time to mitigate**, rather than time to deeply root-cause. Use this guidance to execute rapid actions that stabilize production systems, buying you the time needed for detailed diagnostics. ## Core Philosophy: Duct-Tape Outage Resolution 1. **Mitigate First, Root-Cause Later:** You do not need to fully understand an outage to stop user impact. 1. **Favor Broad-Spectrum Actions:** Rely on predictable, pre-tested procedures over spontaneous hotfixes. 1. **Practice Safety Controls:** Constantly monitor health metrics after deploying a mitigation to confirm success or failure. ## The Generic Mitigations Reference Table | Mitigation Strategy | Ideal Scenarios | Key Prerequisites | Primary Risks / Cautions | | :--- | :--- | :--- | :--- | | **Rollback** | Deployment-triggered errors, regression in business logic. | A clearly demarcated known-good build. *Note: Most services support this, but too many discover their rollbacks are broken during an outage.* | State incompatibilities (schema changes, API version mismatches). | | **Data Rollback** | Corrupt pipeline builds, stale data, bad configurations. Highly useful for content-heavy services. | Frequent, tested backups and decoupled state. | Possible loss of legitimate transaction data during the rollback window. | | **Degrade** | Capacity saturation, cascade failures, system-wide load. | Toggles to turn off expensive/non-essential workflows. | Reduced functionality for end-users. *Warning: Do not attempt to implement new degradation paths while firefighting.* | | **Upsize** | Unexpected traffic spikes, resource starvation. | Scalable backing infra. It is expensive but avoids outages. | Resource exhaustion further downstream. *Note: Scaling is complex; adding replicas may shift bottlenecks.* | | **Block List** | Single disruptive tenant, "Query of Death", DoS attack. | Quick filtering rules (API Gateway, WAF, proxy). | Over-blocking legitimate users if filters are too broad. | | **Drain** | Localized infrastructure failures, regional blackouts. | Multi-homed environments capable of absorbing rerouted traffic. | Overloading the healthy secondary region, cascading failures. | | **Quarantine** | Hot DB rows, spammy users, poisoned traffic streams. | Ability to separate logical usage streams instantly. | Complex orchestration; might only delay cascading side effects. | ## Model Instruction: Applying Mitigations When dealing with a live incident: 1. **Classify symptoms immediately:** Do not jump to debugger tools. Identify the blast radius. Is it localized to one region? Is it tied to a recent push? 1. **Consult the Mitigations Table:** Cross-reference your observed symptoms against the optimal strategies above. 1. **Draft the Execution Steps:** Formulate a structured sequence (e.g., *Drain us-east1 -> Confirm load balances -> Evaluate latency*). 1. **Evaluate Constraints:** Check if state transitions (e.g., database schema) prohibit rollbacks. *Remember: An unfamiliar mitigation is a hazard. Rely primarily on tools you have exercised.* ## References - **Generic Mitigations: A Philosophy of Duct-Tape Outage Resolution** by Jennifer Mace: [Google SRE Classroom](https://sre.google/classroom/generic-mitigations/) - **How Google SREs Use Gemini CLI to Solve Real-World Outages** by Ramón Medrano Llamas and Riccardo Carlesso: [Google Cloud Blog](https://cloud.google.com/blog/topics/developers-practitioners/how-google-sres-use-gemini-cli-to-solve-real-world-outages?e=48754805)