theinterneti Incident Response Skill
Overview
Emergency workflow for handling production incidents with DevOps Engineer and Observability Expert.
Duration: Variable (15 minutes - 2 hours) Personas Required: DevOps, Observability
Incident Workflow
Phase 1: Detect (@observability-expert)
Actions:
- Receive alert from monitoring
- Check Grafana dashboards
- Query Prometheus for metrics
- Assess impact and severity
Phase 2: Investigate (@devops-engineer + @observability-expert)
Actions:
- Review recent deployments
- Check application logs
- Analyze distributed traces
- Identify root cause
Phase 3: Mitigate (@devops-engineer)
Actions:
- Rollback deployment if needed
- Scale resources if capacity issue
- Apply hotfix if minor bug
- Notify stakeholders
Phase 4: Resolve (@devops-engineer)
Actions:
- Verify metrics return to normal
- Close alerts
- Monitor for 30 minutes
- Document actions
Phase 5: Postmortem (@observability-expert)
Actions:
- Write incident report in Logseq
- Identify preventive measures
- Update runbooks
- Share learnings with team
Success Criteria
- ✅ Incident detected <5 minutes
- ✅ Response initiated <10 minutes
- ✅ Service restored <1 hour
- ✅ Postmortem complete <24 hours
- ✅ Preventive measures implemented