Incident Response
This is an emergency procedure. Follow steps carefully.
Steps
1. Assess
- What is the nature of the incident? (outage, security breach, data corruption)
- When did it start?
- What is the blast radius? (single user, all users, data integrity)
2. Contain
- If security breach: rotate all API keys and secrets immediately
- If outage: check health endpoint and CI status
- If data corruption: stop the service, take a backup immediately
# Take immediate backup (adjust for your database)
# SQLite
cp database.db database.db.incident-backup
# PostgreSQL
pg_dump dbname > incident-backup.sql
3. Diagnose
# Check service status
curl -sf https://{PRODUCTION_URL}/health
# Check recent CI
gh run list --limit 5
# Check recent commits for the cause
git log --oneline -10
4. Remediate
Follow the relevant procedure from the project's runbook or operational docs.
5. Recover
- Verify service is healthy
- Verify data integrity
- Notify affected users if applicable
6. Document
Create an incident report with:
- Timeline of events
- Root cause
- Resolution steps taken
- Prevention measures for the future