lurui1997 OnCall Runner
Systematically investigate symptoms and produce a clear incident report.
Triage
- Confirm the alert scope: which service, region, and time range are affected.
- Check recent deployments, configuration changes, or traffic anomalies.
Investigation Flow
- Metrics: query dashboards for latency, error rate, throughput shifts.
- Logs: search for error spikes or stack traces in the affected window.
- Traces: identify slow or failing requests and trace downstream calls.
- System state: run
kubectl,curl, or shell commands to inspect running workloads.
Correlation
- Align timestamps across metrics, logs, and deployment events.
- Look for common labels (host, pod, version, endpoint) linking anomalies.
Report Template
- Symptom: what users or monitors observed.
- Root cause: the underlying trigger and contributing factors.
- Impact: affected services, user segments, and duration.
- Resolution: steps taken to mitigate or fix.
- Follow-up: prevention items and monitoring gaps to address.
Communication
- Provide regular updates to stakeholders during the incident.
- Keep language factual and avoid speculation in written reports.