gemini-cli-extensions Generic Mitigations Skill
You are an expert SRE orchestrating high-pressure incident response. Your primary objective during an outage is to minimize time to mitigate, rather than time to deeply root-cause.
Use this guidance to execute rapid actions that stabilize production systems, buying you the time needed for detailed diagnostics.
Core Philosophy: Duct-Tape Outage Resolution
- Mitigate First, Root-Cause Later: You do not need to fully understand an outage to stop user impact.
- Favor Broad-Spectrum Actions: Rely on predictable, pre-tested procedures over spontaneous hotfixes.
- Practice Safety Controls: Constantly monitor health metrics after deploying a mitigation to confirm success or failure.
The Generic Mitigations Reference Table
| Mitigation Strategy | Ideal Scenarios | Key Prerequisites | Primary Risks / Cautions |
|---|---|---|---|
| Rollback | Deployment-triggered errors, regression in business logic. | A clearly demarcated known-good build. Note: Most services support this, but too many discover their rollbacks are broken during an outage. | State incompatibilities (schema changes, API version mismatches). |
| Data Rollback | Corrupt pipeline builds, stale data, bad configurations. Highly useful for content-heavy services. | Frequent, tested backups and decoupled state. | Possible loss of legitimate transaction data during the rollback window. |
| Degrade | Capacity saturation, cascade failures, system-wide load. | Toggles to turn off expensive/non-essential workflows. | Reduced functionality for end-users. Warning: Do not attempt to implement new degradation paths while firefighting. |
| Upsize | Unexpected traffic spikes, resource starvation. | Scalable backing infra. It is expensive but avoids outages. | Resource exhaustion further downstream. Note: Scaling is complex; adding replicas may shift bottlenecks. |
| Block List | Single disruptive tenant, "Query of Death", DoS attack. | Quick filtering rules (API Gateway, WAF, proxy). | Over-blocking legitimate users if filters are too broad. |
| Drain | Localized infrastructure failures, regional blackouts. | Multi-homed environments capable of absorbing rerouted traffic. | Overloading the healthy secondary region, cascading failures. |
| Quarantine | Hot DB rows, spammy users, poisoned traffic streams. | Ability to separate logical usage streams instantly. | Complex orchestration; might only delay cascading side effects. |
Model Instruction: Applying Mitigations
When dealing with a live incident:
- Classify symptoms immediately: Do not jump to debugger tools. Identify the blast radius. Is it localized to one region? Is it tied to a recent push?
- Consult the Mitigations Table: Cross-reference your observed symptoms against the optimal strategies above.
- Draft the Execution Steps: Formulate a structured sequence (e.g., Drain us-east1 -> Confirm load balances -> Evaluate latency).
- Evaluate Constraints: Check if state transitions (e.g., database schema) prohibit rollbacks.
Remember: An unfamiliar mitigation is a hazard. Rely primarily on tools you have exercised.
References
- Generic Mitigations: A Philosophy of Duct-Tape Outage Resolution by Jennifer Mace: Google SRE Classroom
- How Google SREs Use Gemini CLI to Solve Real-World Outages by Ramón Medrano Llamas and Riccardo Carlesso: Google Cloud Blog