Incident Response
This skill routes to the incident-responder agent — a senior incident commander and SRE specialist (Opus) that diagnoses production incidents methodically, mitigates impact quickly, and produces thorough post-mortems.
Severity Classification
| Level |
Definition |
Response Time |
Examples |
| SEV1 |
Service down, data loss risk |
Immediate (< 15 min) |
API unreachable, database corruption, security breach |
| SEV2 |
Major feature broken, significant user impact |
< 30 min |
Auth failing, payments broken, Centrifugo down |
| SEV3 |
Degraded performance or minor feature broken |
< 2 hours |
Slow API, Sidekiq backed up, partial outage |
| SEV4 |
Minor issue, workaround exists |
Next business day |
UI glitch, non-critical job failing |
What the Agent Does
The incident-responder agent has access to Read, Grep, Glob, and Bash tools for live diagnosis:
- Triage — checks health endpoints, ECS task status, recent deployments, CloudWatch metrics
- Component identification — systematically checks Rails (ECS), PostgreSQL/PostGIS (RDS), Redis/Sidekiq, Centrifugo
- Mitigation — ECS rollback, migration rollback, query termination, Sidekiq worker restart
- Communication — generates incident notifications and status updates
- Post-mortem — produces structured post-mortem documents with timeline, root cause, action items
Escalation Path
| Level |
Who |
When |
| L1 |
On-call engineer |
SEV3/SEV4 — first 30 minutes |
| L2 |
Tech lead + on-call |
SEV2 or L1 unresolved within 30 min |
| L3 |
CTO + full team |
SEV1 or data loss risk |
Chaos Engineering
The agent also supports proactive resilience testing:
- GameDay experiment design (hypothesis, steady state, abort conditions)
- Fault injection patterns (network, compute, storage, dependency failures)
- Resilience checklist validation
See agents/incident-responder.md for the full diagnosis protocol, communication templates, and chaos engineering procedures.
See docs/runbooks/ for component-specific operational runbooks.
Deep guides (read on demand, do not preload)
- Runbooks: database connection exhaustion, Redis OOM, Sidekiq queue backup, ECS crash loops, Centrifugo disconnections, PostGIS query timeouts →
references/runbooks.md
1---2name: incident-response3description: Guide incident response for production issues including diagnosis, mitigation, communication, and post-mortem analysis. Use this skill whenever someone reports an outage, performance degradation, production error, or says things like "production is down", "users are reporting errors", "the API is slow", "we have an incident", "something broke in prod", "check the logs for errors", or "we need a post-mortem". Also trigger when someone mentions SEV1/SEV2 classification, incident communication, root cause analysis, or wants to write a post-mortem document.4---56# Incident Response78This skill routes to the **incident-responder** agent — a senior incident commander and SRE specialist (Opus) that diagnoses production incidents methodically, mitigates impact quickly, and produces thorough post-mortems.910## Severity Classification1112| Level | Definition | Response Time | Examples |13|-------|-----------|---------------|---------|14| **SEV1** | Service down, data loss risk | Immediate (< 15 min) | API unreachable, database corruption, security breach |15| **SEV2** | Major feature broken, significant user impact | < 30 min | Auth failing, payments broken, Centrifugo down |16| **SEV3** | Degraded performance or minor feature broken | < 2 hours | Slow API, Sidekiq backed up, partial outage |17| **SEV4** | Minor issue, workaround exists | Next business day | UI glitch, non-critical job failing |1819## What the Agent Does2021The incident-responder agent has access to **Read, Grep, Glob, and Bash** tools for live diagnosis:22231. **Triage** — checks health endpoints, ECS task status, recent deployments, CloudWatch metrics242. **Component identification** — systematically checks Rails (ECS), PostgreSQL/PostGIS (RDS), Redis/Sidekiq, Centrifugo253. **Mitigation** — ECS rollback, migration rollback, query termination, Sidekiq worker restart264. **Communication** — generates incident notifications and status updates275. **Post-mortem** — produces structured post-mortem documents with timeline, root cause, action items2829## Escalation Path3031| Level | Who | When |32|-------|-----|------|33| L1 | On-call engineer | SEV3/SEV4 — first 30 minutes |34| L2 | Tech lead + on-call | SEV2 or L1 unresolved within 30 min |35| L3 | CTO + full team | SEV1 or data loss risk |3637## Chaos Engineering3839The agent also supports proactive resilience testing:40- GameDay experiment design (hypothesis, steady state, abort conditions)41- Fault injection patterns (network, compute, storage, dependency failures)42- Resilience checklist validation4344See `agents/incident-responder.md` for the full diagnosis protocol, communication templates, and chaos engineering procedures.45See `docs/runbooks/` for component-specific operational runbooks.4647## Deep guides (read on demand, do not preload)4849- Runbooks: database connection exhaustion, Redis OOM, Sidekiq queue backup, ECS crash loops, Centrifugo disconnections, PostGIS query timeouts → `references/runbooks.md`
Kaakati