SRE Incident Responder
Overview
Diagnose first, mutate last. Establish the blast radius, timeline, and root cause from evidence before touching anything.
Signal sources
Cross-platform (kubectl, k8sgpt, and docker work identically on Linux, macOS, and Windows):
kubectl get events -A --sort-by=.lastTimestamp
kubectl logs <pod> -n <ns> --previous --tail=200
kubectl top pods -n <ns>
k8sgpt analyze --explain # if k8sgpt is installed
docker logs --tail 200 <container>
Host logs per OS:
journalctl -u <service> --since "1 hour ago" # Linux
log show --last 1h --predicate 'process == "<service>"' # macOS
Get-WinEvent -LogName Application -MaxEvents 100 |
Where-Object LevelDisplayName -in 'Error','Critical' # Windows
Also use Grafana/Prometheus MCP servers when configured for dashboards, alert state, and metric queries.
Workflow
- Scope: what is broken, since when, for whom. Pin the first bad timestamp.
- Correlate across layers: recent deploys/config changes first (most incidents are changes), then resources (OOM, CPU throttling, disk), then dependencies (DB, DNS, upstream APIs), then infra.
- State the root cause with the evidence chain, not just the symptom.
- Propose remediation in two parts: immediate mitigation (rollback, restart, scale) and durable fix (code/config/capacity).
- Apply mitigation only with approval; verify recovery against the original symptom.
- Write a short postmortem: timeline, root cause, impact, actions, follow-ups. Save it under
ops/incidents/.
Rules
- Rollbacks and restarts on prod require
human-approval.
- Never delete evidence (logs, crashed pods) before capturing it.
- If the cause cannot be proven, list the ranked hypotheses and the test for each; do not guess-remediate.
1---2name: sre-incident-responder3description: Triage incidents like an SRE - correlate logs, metrics, events and traces, find the root cause, propose remediation, and write the postmortem. Uses k8sgpt when available.4---56# SRE Incident Responder78## Overview910Diagnose first, mutate last. Establish the blast radius, timeline, and root cause from evidence before touching anything.1112## Signal sources1314Cross-platform (kubectl, k8sgpt, and docker work identically on Linux, macOS, and Windows):1516```bash17kubectl get events -A --sort-by=.lastTimestamp18kubectl logs <pod> -n <ns> --previous --tail=20019kubectl top pods -n <ns>20k8sgpt analyze --explain # if k8sgpt is installed21docker logs --tail 200 <container>22```2324Host logs per OS:2526```bash27journalctl -u <service> --since "1 hour ago" # Linux28log show --last 1h --predicate 'process == "<service>"' # macOS29```3031```powershell32Get-WinEvent -LogName Application -MaxEvents 100 |33 Where-Object LevelDisplayName -in 'Error','Critical' # Windows34```3536Also use Grafana/Prometheus MCP servers when configured for dashboards, alert state, and metric queries.3738## Workflow39401. Scope: what is broken, since when, for whom. Pin the first bad timestamp.412. Correlate across layers: recent deploys/config changes first (most incidents are changes), then resources (OOM, CPU throttling, disk), then dependencies (DB, DNS, upstream APIs), then infra.423. State the root cause with the evidence chain, not just the symptom.434. Propose remediation in two parts: immediate mitigation (rollback, restart, scale) and durable fix (code/config/capacity).445. Apply mitigation only with approval; verify recovery against the original symptom.456. Write a short postmortem: timeline, root cause, impact, actions, follow-ups. Save it under `ops/incidents/`.4647## Rules4849- Rollbacks and restarts on prod require `human-approval`.50- Never delete evidence (logs, crashed pods) before capturing it.51- If the cause cannot be proven, list the ranked hypotheses and the test for each; do not guess-remediate.
Navinspire-ia