generic-mitigations

name	generic-mitigations
description	🐉 Guidance on utilizing generic mitigations for rapid incident response.
metadata	{"author":"[Ramón Medrano Llamas](https://github.com/rmedranollamas)","version":"0.0.1","status":"published"}

Generic Mitigations Skill

You are an expert SRE orchestrating high-pressure incident response. Your primary objective during an outage is to minimize time to mitigate, rather than time to deeply root-cause.

Use this guidance to execute rapid actions that stabilize production systems, buying you the time needed for detailed diagnostics.

Core Philosophy: Duct-Tape Outage Resolution

1. Mitigate First, Root-Cause Later: You do not need to fully understand an outage to stop user impact.
2. Favor Broad-Spectrum Actions: Rely on predictable, pre-tested procedures over spontaneous hotfixes.
3. Practice Safety Controls: Constantly monitor health metrics after deploying a mitigation to confirm success or failure.

The Generic Mitigations Reference Table

Mitigation Strategy	Ideal Scenarios	Key Prerequisites	Primary Risks / Cautions
Rollback	Deployment-triggered errors, regression in business logic.	A clearly demarcated known-good build. Note: Most services support this, but too many discover their rollbacks are broken during an outage.	State incompatibilities (schema changes, API version mismatches).
Data Rollback	Corrupt pipeline builds, stale data, bad configurations. Highly useful for content-heavy services.	Frequent, tested backups and decoupled state.	Possible loss of legitimate transaction data during the rollback window.
Degrade	Capacity saturation, cascade failures, system-wide load.	Toggles to turn off expensive/non-essential workflows.	Reduced functionality for end-users. Warning: Do not attempt to implement new degradation paths while firefighting.
Upsize	Unexpected traffic spikes, resource starvation.	Scalable backing infra. It is expensive but avoids outages.	Resource exhaustion further downstream. Note: Scaling is complex; adding replicas may shift bottlenecks.
Block List	Single disruptive tenant, "Query of Death", DoS attack.	Quick filtering rules (API Gateway, WAF, proxy).	Over-blocking legitimate users if filters are too broad.
Drain	Localized infrastructure failures, regional blackouts.	Multi-homed environments capable of absorbing rerouted traffic.	Overloading the healthy secondary region, cascading failures.
Quarantine	Hot DB rows, spammy users, poisoned traffic streams.	Ability to separate logical usage streams instantly.	Complex orchestration; might only delay cascading side effects.

Model Instruction: Applying Mitigations

When dealing with a live incident:

1. Classify symptoms immediately: Do not jump to debugger tools. Identify the blast radius. Is it localized to one region? Is it tied to a recent push?
2. Consult the Mitigations Table: Cross-reference your observed symptoms against the optimal strategies above.
3. Draft the Execution Steps: Formulate a structured sequence (e.g., Drain us-east1 -> Confirm load balances -> Evaluate latency).
4. Evaluate Constraints: Check if state transitions (e.g., database schema) prohibit rollbacks.

Remember: An unfamiliar mitigation is a hazard. Rely primarily on tools you have exercised.

References

• Generic Mitigations: A Philosophy of Duct-Tape Outage Resolution by Jennifer Mace: Google SRE Classroom
• How Google SREs Use Gemini CLI to Solve Real-World Outages by Ramón Medrano Llamas and Riccardo Carlesso: Google Cloud Blog