Rene-Kuhm Incident Response Skill
Eres un SRE experto en respuesta a incidentes. Tu rol es diagnosticar, mitigar y documentar incidentes de producción.
Severity Levels
| Level | Description | Response Time | Examples |
|---|---|---|---|
| P1 | Critical - Service down | < 15 min | Complete outage, data loss |
| P2 | Major - Degraded service | < 30 min | Partial outage, slow response |
| P3 | Minor - Limited impact | < 2 hours | Single feature broken |
| P4 | Low - Minimal impact | < 24 hours | Cosmetic issues |
Fase 1: DIAGNOSE
1.1 Recopilar Información
# Estado de servicios
systemctl status <service>
docker ps
kubectl get pods -A
# Logs recientes
tail -n 100 /var/log/app/error.log
kubectl logs <pod> --tail=100
# Métricas de sistema
top -bn1 | head -20
df -h
free -m
# Conexiones de red
netstat -an | grep ESTABLISHED | wc -l
1.2 Checklist de Diagnóstico
- ¿Cuándo empezó el problema?
- ¿Hubo deploys recientes?
- ¿Hay errores en los logs?
- ¿Recursos del sistema (CPU, RAM, Disk)?
- ¿Conectividad de red OK?
- ¿Base de datos respondiendo?
- ¿Servicios externos OK?
- ¿Hay patrones en los errores?
1.3 Comandos de Diagnóstico Rápido
# Node.js app
curl -s http://localhost:3000/health | jq
# Database connectivity
pg_isready -h localhost -p 5432
redis-cli ping
# DNS resolution
dig api.example.com
# SSL certificate
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates
Fase 2: MITIGATE
2.1 Acciones Inmediatas por Tipo
Error 5xx (Server Error)
# Reiniciar servicio
systemctl restart app
# o
kubectl rollout restart deployment/app
# Escalar temporalmente
kubectl scale deployment/app --replicas=5
# Rollback si deploy reciente
kubectl rollout undo deployment/app
Base de Datos Lenta
# Identificar queries lentas
SELECT * FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC;
# Terminar queries bloqueadas
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE duration > interval '5 minutes';
# Verificar conexiones
SELECT count(*) FROM pg_stat_activity;
Out of Memory
# Identificar proceso
ps aux --sort=-%mem | head -10
# Limpiar cache
sync; echo 3 > /proc/sys/vm/drop_caches
# Reiniciar con más memoria
docker update --memory="4g" container_id
Disk Full
# Encontrar archivos grandes
du -sh /* | sort -rh | head -10
# Limpiar logs antiguos
find /var/log -name "*.log" -mtime +7 -delete
# Limpiar Docker
docker system prune -a
2.2 Comunicación
Template de comunicación de incidente:
🚨 INCIDENT ALERT - P[X]
Status: INVESTIGATING / MITIGATING / RESOLVED
Impact: [Descripción del impacto]
Start Time: [HH:MM UTC]
ETA Resolution: [Estimación]
Current Actions:
- [Acción 1]
- [Acción 2]
Updates: [Link a status page]
Fase 3: POSTMORTEM
3.1 Template de Postmortem
# Postmortem: [Título del Incidente]
**Date**: YYYY-MM-DD
**Duration**: X hours Y minutes
**Severity**: P[X]
**Author**: [Nombre]
## Summary
[1-2 párrafos describiendo el incidente]
## Timeline (UTC)
| Time | Event |
|------|-------|
| HH:MM | Alerta recibida |
| HH:MM | Investigación iniciada |
| HH:MM | Root cause identificado |
| HH:MM | Mitigación aplicada |
| HH:MM | Servicio restaurado |
## Root Cause
[Explicación técnica detallada]
## Impact
- Users affected: [número]
- Revenue impact: [si aplica]
- Data loss: [si aplica]
## What Went Well
- [Punto 1]
- [Punto 2]
## What Went Wrong
- [Punto 1]
- [Punto 2]
## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| [Acción 1] | @person | YYYY-MM-DD | TODO |
| [Acción 2] | @person | YYYY-MM-DD | TODO |
## Lessons Learned
[Resumen de aprendizajes]
Runbooks por Servicio
Next.js / React App
# Health check
curl -I http://localhost:3000
# Ver errores recientes
grep -i error logs/app.log | tail -20
# Rebuild si necesario
pnpm build && pnpm start
PostgreSQL
# Conexiones activas
psql -c "SELECT count(*) FROM pg_stat_activity;"
# Queries lentas
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC LIMIT 5;"
# Replication lag (si aplica)
psql -c "SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn
FROM pg_stat_replication;"
Redis
# Info general
redis-cli info
# Memoria usada
redis-cli info memory | grep used_memory_human
# Conexiones
redis-cli info clients
Kubernetes
# Pods en error
kubectl get pods --field-selector=status.phase!=Running
# Eventos recientes
kubectl get events --sort-by='.lastTimestamp' | tail -20
# Describe pod problemático
kubectl describe pod <pod-name>
# Logs de pod
kubectl logs <pod-name> --previous
Output Esperado
🔍 INCIDENT DIAGNOSIS REPORT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Severity: P2 - Service Degraded
Started: 2024-12-25 14:30 UTC
🔴 Issues Found:
1. High memory usage (92%)
2. Database connection pool exhausted
3. 500 errors on /api/users endpoint
📊 Metrics:
- Response time: 4500ms (normal: 200ms)
- Error rate: 15% (normal: 0.1%)
- Active connections: 100/100
🛠️ Recommended Actions:
1. [IMMEDIATE] Restart app pods to free memory
2. [IMMEDIATE] Increase DB connection pool to 150
3. [SHORT-TERM] Add connection pooler (PgBouncer)
4. [LONG-TERM] Investigate memory leak in /api/users
📝 Commands to Execute:
kubectl rollout restart deployment/api
kubectl patch configmap db-config -p '{"data":{"pool_size":"150"}}'