Incident Response & SRE
You triage by impact, not by panic level. Step one is always: is the system still degraded? If yes, mitigate first, investigate later. Post-mortems are about systems, not people.
When to use
- Production is down or degraded
- Users are reporting errors
- Monitoring alerts are firing
- Need to write a post-mortem
- Creating runbooks for known failure modes
Incident Response Protocol
1. ASSESS (First 5 Minutes)
- What's broken? Check dashboards, error rates, user reports
- Who's affected? All users, specific region, specific feature?
- When did it start? Correlate with recent deploys or changes
- Is it getting worse? Check if error rate is increasing
2. MITIGATE (Stop the Bleeding)
Can you revert the last deploy? → Revert and verify
Is it a database issue? → Check connections, locks, disk space
Is it an Edge Function failure? → Check logs, redeploy known-good version
Is it a third-party outage? → Enable fallback/graceful degradation
Is it a traffic spike? → Scale resources or enable rate limiting
3. INVESTIGATE (After Mitigation)
Use the Supabase MCP tools:
get_logs(service: "edge-function") # Edge Function errors
get_logs(service: "postgres") # Database errors
get_logs(service: "auth") # Authentication issues
get_logs(service: "api") # PostgREST/API gateway
get_advisors(type: "performance") # Performance issues
4. COMMUNICATE
- Internal: What's broken, what we're doing, ETA if known
- Users: Acknowledge the issue, no false promises on timeline
5. RESOLVE & DOCUMENT
- Deploy the fix
- Verify metrics return to normal
- Write a blameless post-mortem
Supabase-Specific Failure Modes
| Symptom |
Check |
Common Cause |
| 500 errors on API calls |
get_logs(service: "api") |
Database connection pool exhausted |
| Edge Function timeout |
get_logs(service: "edge-function") |
Slow query or external API timeout |
| Auth failures (401/403) |
get_logs(service: "auth") |
JWT expired, RLS policy blocking |
| Realtime not updating |
get_logs(service: "realtime") |
Channel not subscribed, RLS on table |
| Storage upload fails |
get_logs(service: "storage") |
Bucket policy, file size limit |
| Slow queries |
get_advisors(type: "performance") |
Missing index, sequential scan |
Post-Mortem Template
## Incident: [Brief Description]
**Date**: YYYY-MM-DD
**Duration**: X hours Y minutes
**Impact**: [Who was affected and how]
**Severity**: P1/P2/P3
### Timeline
- HH:MM — First alert / user report
- HH:MM — Investigation started
- HH:MM — Root cause identified
- HH:MM — Mitigation applied
- HH:MM — Full resolution confirmed
### Root Cause
[What actually went wrong at the system level — no blame, no names]
### What Went Well
- [Things that helped detection/resolution]
### What Could Be Improved
- [Gaps in monitoring, testing, or process]
### Action Items
- [ ] [Specific fix] — Owner — Due date
- [ ] [Monitoring improvement] — Owner — Due date
- [ ] [Process change] — Owner — Due date
Runbook Essentials
A good runbook answers these for every failure mode:
- How will I know? (alert, error message, user report)
- How do I verify? (dashboard, query, health check)
- What do I do? (step-by-step, not "investigate")
- How do I confirm it's fixed? (metric returns to normal, test passes)
- Who do I escalate to? (if steps don't resolve it)
1---2name: incident-response3description: Use for production incident response, outage triage, root cause analysis, blameless post-mortems, runbook creation, and SRE practices. Covers Supabase-specific debugging (Edge Function failures, database issues, auth problems), log analysis, and incident communication.4---56# Incident Response & SRE78You triage by impact, not by panic level. Step one is always: is the system still degraded? If yes, mitigate first, investigate later. Post-mortems are about systems, not people.910## When to use11- Production is down or degraded12- Users are reporting errors13- Monitoring alerts are firing14- Need to write a post-mortem15- Creating runbooks for known failure modes1617## Incident Response Protocol1819### 1. ASSESS (First 5 Minutes)20- **What's broken?** Check dashboards, error rates, user reports21- **Who's affected?** All users, specific region, specific feature?22- **When did it start?** Correlate with recent deploys or changes23- **Is it getting worse?** Check if error rate is increasing2425### 2. MITIGATE (Stop the Bleeding)26```27Can you revert the last deploy? → Revert and verify28Is it a database issue? → Check connections, locks, disk space29Is it an Edge Function failure? → Check logs, redeploy known-good version30Is it a third-party outage? → Enable fallback/graceful degradation31Is it a traffic spike? → Scale resources or enable rate limiting32```3334### 3. INVESTIGATE (After Mitigation)35Use the Supabase MCP tools:36```37get_logs(service: "edge-function") # Edge Function errors38get_logs(service: "postgres") # Database errors39get_logs(service: "auth") # Authentication issues40get_logs(service: "api") # PostgREST/API gateway41get_advisors(type: "performance") # Performance issues42```4344### 4. COMMUNICATE45- **Internal**: What's broken, what we're doing, ETA if known46- **Users**: Acknowledge the issue, no false promises on timeline4748### 5. RESOLVE & DOCUMENT49- Deploy the fix50- Verify metrics return to normal51- Write a blameless post-mortem5253## Supabase-Specific Failure Modes5455| Symptom | Check | Common Cause |56|---|---|---|57| 500 errors on API calls | `get_logs(service: "api")` | Database connection pool exhausted |58| Edge Function timeout | `get_logs(service: "edge-function")` | Slow query or external API timeout |59| Auth failures (401/403) | `get_logs(service: "auth")` | JWT expired, RLS policy blocking |60| Realtime not updating | `get_logs(service: "realtime")` | Channel not subscribed, RLS on table |61| Storage upload fails | `get_logs(service: "storage")` | Bucket policy, file size limit |62| Slow queries | `get_advisors(type: "performance")` | Missing index, sequential scan |6364## Post-Mortem Template6566```markdown67## Incident: [Brief Description]68**Date**: YYYY-MM-DD69**Duration**: X hours Y minutes70**Impact**: [Who was affected and how]71**Severity**: P1/P2/P37273### Timeline74- HH:MM — First alert / user report75- HH:MM — Investigation started76- HH:MM — Root cause identified77- HH:MM — Mitigation applied78- HH:MM — Full resolution confirmed7980### Root Cause81[What actually went wrong at the system level — no blame, no names]8283### What Went Well84- [Things that helped detection/resolution]8586### What Could Be Improved87- [Gaps in monitoring, testing, or process]8889### Action Items90- [ ] [Specific fix] — Owner — Due date91- [ ] [Monitoring improvement] — Owner — Due date92- [ ] [Process change] — Owner — Due date93```9495## Runbook Essentials9697A good runbook answers these for every failure mode:981. **How will I know?** (alert, error message, user report)992. **How do I verify?** (dashboard, query, health check)1003. **What do I do?** (step-by-step, not "investigate")1014. **How do I confirm it's fixed?** (metric returns to normal, test passes)1025. **Who do I escalate to?** (if steps don't resolve it)
henryhawke