manastalukdar Incident Response
Run a structured triage when something is broken in production. Collects symptoms, correlates recent changes, generates diagnostic commands, and drafts an incident report.
Usage
/incident-response # start a new incident triage
/incident-response "<symptom description>" # start with known symptom
/incident-response report # draft final incident report from current session
Behavior
Step 1 — Establish severity and scope
Ask (or infer from $ARGUMENTS):
- What is broken? (service / endpoint / feature)
- Who is affected? (all users / subset / internal only)
- Since when? (approximate start time)
- Is it complete outage or degraded performance?
Assign severity:
- SEV1: Complete outage or data loss risk
- SEV2: Major feature broken, workaround unavailable
- SEV3: Degraded performance or minor feature broken
Step 2 — Correlate recent changes
# Last 20 commits across the affected timeframe
git log --oneline --since="6 hours ago"
# Recently modified config or infra files
git diff HEAD~5..HEAD --name-only | grep -E "\.env|config|deploy|k8s|helm|terraform"
# Any dependency bumps
git diff HEAD~5..HEAD -- package.json package-lock.json requirements.txt go.mod
Flag any commit touching the affected service within the incident window.
Step 3 — Generate diagnostic commands
Based on the symptom, output a runbook of commands to execute:
For HTTP errors (4xx/5xx spike):
# Tail application logs for error patterns
grep -i "error\|exception\|fatal" app.log | tail -50
# Check error rate by endpoint
grep "HTTP/1.1\" [45][0-9][0-9]" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
For latency spike:
# Find slow requests
grep -E "took [0-9]{4,}ms|duration=[0-9]{4,}" app.log | tail -20
# Check DB slow query log
grep "slow query" /var/log/mysql/slow.log 2>/dev/null | tail -20
For memory / CPU:
ps aux --sort=-%mem | head -10
free -h
df -h
For dependency outage:
# Test external health endpoints
curl -s -o /dev/null -w "%{http_code}" https://status.stripe.com/api/v2/status.json
Step 4 — Identify blast radius
Check which other services or features share the affected component:
# Which files import the affected module?
grep -rn "from.*<affected-module>\|require.*<affected-module>" --include="*.ts" --include="*.py" -l .
Step 5 — Propose immediate mitigations
For each confirmed root cause, offer the minimum safe mitigation:
- Recent bad deploy: rollback command (
git revert <sha>, feature flag off) - Dependency outage: enable fallback, disable non-critical feature
- Config change: revert specific key
- Traffic spike: rate-limit endpoint, scale horizontally
Step 6 — Draft incident report
## Incident Report — [TITLE]
**Severity**: SEV[1/2/3]
**Start**: [timestamp]
**End / Ongoing**: [timestamp or ONGOING]
**Affected**: [service/feature/users]
### Timeline
- HH:MM — Symptom first observed
- HH:MM — Alert fired / team notified
- HH:MM — Root cause identified
- HH:MM — Mitigation applied
- HH:MM — Service restored
### Root Cause
[What broke, why it broke]
### Impact
[Users affected, error rate, duration]
### Mitigation
[What was done to restore service]
### Follow-up Actions
- [ ] Long-term fix (link to issue)
- [ ] Add monitoring / alerting for [metric]
- [ ] Add resilience control for [dependency]
- [ ] Update runbook for [scenario]
Edge Cases
- No git history: Skip Step 2 and note it; continue with diagnostics.
- No access to logs: List diagnostic commands for the operator to run; generate report from provided output.
- Ongoing incident: Focus on mitigation steps first; defer full report to Step 6 until service is restored.
Token Optimization
Expected range: 400–1,200 tokens (triage phase); 300–600 tokens (report-only mode)
Patterns used: Bash for system queries (git log, grep), progressive disclosure (diagnostics → root cause → report), early exit (if no symptom provided, ask before proceeding)
Early exit: Running /incident-response report with no prior triage context prompts for the incident summary before generating the report.