SRE Incident Responder
Overview
Diagnose first, mutate last. Establish the blast radius, timeline, and root cause from evidence before touching anything.
Signal sources
Cross-platform (kubectl, k8sgpt, and docker work identically on Linux, macOS, and Windows):
kubectl get events -A --sort-by=.lastTimestamp
kubectl logs <pod> -n <ns> --previous --tail=200
kubectl top pods -n <ns>
k8sgpt analyze --explain # if k8sgpt is installed
docker logs --tail 200 <container>
Host logs per OS:
journalctl -u <service> --since "1 hour ago" # Linux
log show --last 1h --predicate 'process == "<service>"' # macOS
Get-WinEvent -LogName Application -MaxEvents 100 |
Where-Object LevelDisplayName -in 'Error','Critical' # Windows
Also use Grafana/Prometheus MCP servers when configured for dashboards, alert state, and metric queries.
Workflow
- Scope: what is broken, since when, for whom. Pin the first bad timestamp.
- Correlate across layers: recent deploys/config changes first (most incidents are changes), then resources (OOM, CPU throttling, disk), then dependencies (DB, DNS, upstream APIs), then infra.
- State the root cause with the evidence chain, not just the symptom.
- Propose remediation in two parts: immediate mitigation (rollback, restart, scale) and durable fix (code/config/capacity).
- Apply mitigation only with approval; verify recovery against the original symptom.
- Write a short postmortem: timeline, root cause, impact, actions, follow-ups. Save it under
ops/incidents/.
Rules
- Rollbacks and restarts on prod require
human-approval.
- Never delete evidence (logs, crashed pods) before capturing it.
- If the cause cannot be proven, list the ranked hypotheses and the test for each; do not guess-remediate.
1---2name: sre-incident-responder3description: Triage incidents like an SRE - correlate logs, metrics, events and traces, find the root cause, propose remediation, and write the postmortem. Uses k8sgpt when available.4metadata:5 source: https://skillmd.com/skills/navinspire-ia/sre-incident-responder6 navin:7 emoji: 🚨8 category: devops9---1011# SRE Incident Responder1213## Overview1415Diagnose first, mutate last. Establish the blast radius, timeline, and root cause from evidence before touching anything.1617## Signal sources1819Cross-platform (kubectl, k8sgpt, and docker work identically on Linux, macOS, and Windows):2021```bash22kubectl get events -A --sort-by=.lastTimestamp23kubectl logs <pod> -n <ns> --previous --tail=20024kubectl top pods -n <ns>25k8sgpt analyze --explain # if k8sgpt is installed26docker logs --tail 200 <container>27```2829Host logs per OS:3031```bash32journalctl -u <service> --since "1 hour ago" # Linux33log show --last 1h --predicate 'process == "<service>"' # macOS34```3536```powershell37Get-WinEvent -LogName Application -MaxEvents 100 |38 Where-Object LevelDisplayName -in 'Error','Critical' # Windows39```4041Also use Grafana/Prometheus MCP servers when configured for dashboards, alert state, and metric queries.4243## Workflow44451. Scope: what is broken, since when, for whom. Pin the first bad timestamp.462. Correlate across layers: recent deploys/config changes first (most incidents are changes), then resources (OOM, CPU throttling, disk), then dependencies (DB, DNS, upstream APIs), then infra.473. State the root cause with the evidence chain, not just the symptom.484. Propose remediation in two parts: immediate mitigation (rollback, restart, scale) and durable fix (code/config/capacity).495. Apply mitigation only with approval; verify recovery against the original symptom.506. Write a short postmortem: timeline, root cause, impact, actions, follow-ups. Save it under `ops/incidents/`.5152## Rules5354- Rollbacks and restarts on prod require `human-approval`.55- Never delete evidence (logs, crashed pods) before capturing it.56- If the cause cannot be proven, list the ranked hypotheses and the test for each; do not guess-remediate.