vibeeval Incident Response Patterns
Severity Classification
| Level | Tanım | Response Time | Escalation |
|---|---|---|---|
| SEV-1 | Tam kesinti, data loss | 5 dk | Tüm ekip |
| SEV-2 | Major feature bozuk | 30 dk | On-call + lead |
| SEV-3 | Minor feature bozuk | 4 saat | On-call |
| SEV-4 | Kozmetik, low impact | Sonraki iş günü | Ticket |
Incident Response Protocol
1. DETECT → Alert tetiklendi / kullanıcı bildirdi
2. TRIAGE → Severity belirle, incident channel aç
3. ASSIGN → Incident Commander + Responder ata
4. CONTAIN → Bleeding'i durdur (rollback, feature flag off)
5. FIX → Root cause'u düzelt
6. VERIFY → Fix'i doğrula (monitoring, smoke test)
7. RESOLVE → Incident kapat, stakeholder bilgilendir
8. REVIEW → Post-incident review (48 saat içinde)
Runbook Template
# Runbook: [Service Name] - [Issue Type]
## Symptoms
- Alert: [alert name]
- Dashboard: [link]
- Expected: [normal behavior]
- Actual: [observed behavior]
## Diagnosis Steps
1. Check [metric/log/dashboard]
2. Run: `kubectl get pods -n production`
3. Check: `SELECT count(*) FROM errors WHERE ...`
## Mitigation
1. Quick fix: [rollback/restart/scale]
2. Feature flag: [disable feature X]
3. Redirect: [failover to backup]
## Escalation
- L1: [on-call engineer]
- L2: [team lead]
- L3: [CTO/VP Eng]
Root Cause Analysis
5 Whys
Problem: API 500 hatası
Why 1: Database connection timeout
Why 2: Connection pool tükendi
Why 3: Slow query connection'ları tutuyor
Why 4: Missing index on frequently queried column
Why 5: Schema change review process eksik
→ Action: Schema change PR'da EXPLAIN ANALYZE zorunlu
Post-Incident Review Template
## Incident Summary
- Duration: [start - end]
- Impact: [affected users/revenue]
- Severity: SEV-[X]
## Timeline
- HH:MM - Alert triggered
- HH:MM - Incident declared
- HH:MM - Root cause identified
- HH:MM - Fix deployed
- HH:MM - Incident resolved
## Root Cause
[Detailed technical explanation]
## Action Items
- [ ] [Fix] Add missing index (owner: @dev, due: date)
- [ ] [Prevent] Add schema review step (owner: @lead)
- [ ] [Detect] Add connection pool alert (owner: @sre)
Checklist
- Severity matrix tanımlı
- On-call rotation aktif
- Runbook'lar güncel
- Incident channel template hazır
- Post-incident review 48 saat içinde
- Action items tracked (Jira/Linear)
- Blameless culture enforced
- Alerting threshold'lar doğru
Anti-Patterns
- Blame culture (kişi değil, sistem düzelt)
- Runbook'suz servis
- Post-incident review atlamak
- Alert fatigue (çok fazla false positive)
- Hotfix without rollback plan