# Incident Investigation

> Use when investigating a production incident, anomaly, or error using observability tooling (Grafana, Sentry, Datadog, New Relic — via MCP or a plain API key) — from first symptom to a confirmed, evidence-backed root cause. Trigger on symptoms like an error-rate spike, latency increase, a failed alert, OOM/restart loops, or "something broke in production but I don't know where or when."

- Skill: `gabrielras/incident-investigation` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add gabrielras/incident-investigation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/gabrielras/incident-investigation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: gabrielras (https://skillmd.com/u/gabrielras)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/gabrielras/incident-investigation

---


# Incident Investigation

## Overview

This is a purely investigative skill: gather evidence from observability tooling, cross-reference it against the actual codebase, and arrive at a confirmed root cause backed by that evidence. It does not write tests, run code, implement a fix, or deploy anything — the deliverable is a root cause report with its evidence trail, handed off to whatever fixes code next.

**Core principle:** monitoring answers questions you already knew to ask (known-unknowns); observability lets you ask a question you didn't pre-build an alert for (unknown-unknowns). Most real incidents worth investigating are unknown-unknowns — reaching for the one dashboard you already have is monitoring behavior applied to a problem that needs observability.

## The Iron Law

```
NO ROOT CAUSE CLAIM WITHOUT VERIFICATION AGAINST REAL EVIDENCE
```

A hypothesis is not a conclusion. Before stating a root cause, cross-check it against the observability evidence *and* the actual code — both have to agree, not just sound plausible together.

## Requires

Works with an MCP server for Grafana, Sentry, Datadog, or New Relic (all ship official servers as of 2026) — **or** with just an API key/token for any of these, no MCP needed. If you have neither an MCP connection nor an API key for the platform in question, say that explicitly rather than guessing what a dashboard "probably" shows.

**Discover before using.** Before calling any platform tool by name, check what's actually connected and available in this session — call the MCP server's own tool listing (`list_tools` or equivalent) rather than assuming. Tool names in this skill and in `references/tool-reference.md` are examples of the *kind* of capability to look for (query a time-series metric, search issues by error signature, correlate a deploy against an incident) — verified as of 2026, but these are actively evolving vendor products; the exact name may have changed since. Match by what the tool does, confirmed against what's actually available right now, not by pattern-matching the name written here.

## Intake — Understand the Report First

Before querying anything, get what the reporter already knows — don't start blind when a human already has half the evidence. Ask (or extract from what was already said):

- **What was observed**, in their words — an error message, a screenshot, a customer complaint, an alert notification
- **Since when** — an exact time if they have one, otherwise "since roughly when" (this seeds the incident window for every path below)
- **Scope** — one user, one customer, one region, or everyone? A specific endpoint/service, or "the whole app"?
- **What's already been tried** — a restart, a rollback, a config change? If someone already acted, that action itself is now part of the timeline, not a clean baseline
- **Any evidence already in hand** — a log line, a trace ID, a request ID, a stack trace pasted into chat. Use it directly instead of re-deriving it from scratch.

A vague report ("it's broken") is still enough to start — go to Path D. Don't block on a perfect report; the paths below are exactly how you fill in what's missing. But don't skip asking, either — a report that already names the service and the approximate time turns a long investigation into a short one.

## Triage by Primary Symptom

Pick the path that matches what was reported. Don't run every path — the symptom tells you where to start.

| Symptom | Start at |
|---|---|
| Error rate / exception spike, alert fired on errors | Path A |
| Slow endpoint, elevated latency, "it's slow" | Path B |
| OOM kill, restart loop, CPU/memory exhaustion | Path C |
| Vague report, no specific symptom yet | Path D |

## Path A — Error Rate Spike

1. **Confirm the window and blast radius.**
   - Grafana: `query_prometheus` with `rate(http_requests_total{status=~"5.."}[5m])` — or `curl /api/ds/query` (see tool-reference.md)
   - Sentry: `search_issues` filtered to the suspected window — or `curl .../issues/?statsPeriod=24h`
   - Datadog: `search_datadog_error_tracking_issues` — or `curl /api/v2/logs/events/search` with `status:error`
   - New Relic: `list_recent_issues` or `execute_nrql_query` with `SELECT count(*) FROM TransactionError SINCE ...`
2. **Find the error pattern, not just the count.** Grafana's Sift tool `find_error_pattern_logs` groups recurring error signatures automatically — built specifically for this (Grafana Cloud; not confirmed on self-hosted, see tool-reference.md). Sentry: `get_event_stacktrace` and `get_issue_breadcrumbs` for the full picture, or `analyze_issue_with_seer` to let Sentry's own AI triage it first. Datadog: `analyze_datadog_logs`. New Relic: `analyze_entity_logs` or `list_entity_error_groups`.
3. Continue to **Reconstruct the Timeline** below once you have: the error signature, the exact time it started, and which service/endpoint it's on.

## Path B — Latency / Slow Endpoint

1. **Confirm which service, not just "it's slow."** RED per service (rate, errors, duration) narrows to the offending service before you look inside it.
2. **Find the slow spans.** Grafana Sift: `find_slow_requests`. Datadog: `apm_latency_bottleneck_summary` or `apm_query_trace`. Sentry: `get_trace_details` and `get_span_details`. New Relic: `analyze_transactions` or `analyze_golden_metrics`.
3. **Check the resource behind it** (Path C, step 1) — a slow endpoint is often a starved resource wearing a latency costume.
4. Continue to **Reconstruct the Timeline** once you have: the specific slow span (DB call, external API, lock wait) and when it started being slow, not just that it currently is.

## Path C — Resource Exhaustion (OOM, Restarts, CPU)

1. **Pull the USE signals for the resource** (utilization, saturation, errors) — CPU/memory over the suspected window. Grafana: `query_prometheus` for the container/pod metrics. Datadog: `get_datadog_metric`. New Relic: `analyze_golden_metrics`.
2. **Correlate with restart/kill events**, not just the metric shape — `get_annotations` (Grafana), `list_change_events` (New Relic), or the platform's own event log often marks the OOM kill or restart directly.
3. Continue to **Reconstruct the Timeline** once you have: the resource that exhausted, the exact time, and whether it's a leak (climbs steadily) or a spike (sudden jump tied to one request/deploy).

## Path D — No Specific Symptom Yet

1. **Sweep the Four Golden Signals** (latency, traffic, errors, saturation) across the suspected time range first — this tells you which of Paths A/B/C actually applies before you commit to one.
2. Once one signal stands out, jump to the matching path above.

## When the Symptom and the Cause Are in Different Services

Any path above can dead-end at "this service is slow/erroring, but its own code looks fine" — that's the signal to check whether the real cause is downstream, in a service this one calls.

1. **Follow the trace ID across the service boundary, not just within one service.** A distributed trace is a tree: a shared trace ID spans every service touched by one request, and each operation is a span with a parent-child link to the span that called it (Grafana Tempo, Sentry's `get_trace_details`/`get_span_details`, Datadog's `apm_query_trace`, and New Relic's distributed tracing all expose this). The parent-child chain across service boundaries is what tells you *where in the call chain* the slowness or error actually started, instead of just confirming it was visible where you first looked.
2. **Use the platform's dependency-graph tool if one is connected**, rather than manually chasing calls service by service — Datadog's `search_datadog_service_dependencies` and New Relic's `list_related_entities` exist specifically for this.
3. **Propagation only works if every hop actually forwards the trace context** (commonly the `traceparent` HTTP header, per the W3C Trace Context standard) — if a service in the chain doesn't propagate it, the trace breaks there, and that break point is itself useful evidence: it's either where instrumentation is missing, or a real place to keep looking manually. See `references/methodology.md` #7.

## Reconstruct the Timeline

Applies after any path above, before moving to root cause:

1. **Write it down as you go** — what you queried, what it showed, when. Not from memory afterward.
2. If a deploy/annotation lines up with the incident start, that's a *lead*, not proof. New Relic's `analyze_deployment_impact` and Sentry's `find_releases` correlate this directly instead of you eyeballing two timestamps — use them before concluding a deploy caused it.
3. **If timestamps from two different tools don't quite agree, that's expected, not necessarily a red flag by itself** — independent systems don't share a clock, and logs/metrics/traces are typically ingested with different delay. Prefer a shared trace ID to line events up across tools when one is available, rather than trying to reconcile wall-clock timestamps down to the second. See `references/methodology.md` #8.
4. Report what you actually observed, including your own missteps in the investigation so far — an edited timeline is a false one, and the next person investigating a similar incident reads this one first.

## Root Cause: From Evidence to Code

The observability evidence above tells you *that* something broke and roughly *where*. This section finds *why*, by reading — not running — the code.

1. **Trace the evidence to an actual code path.** The stack trace, span, or log line points at specific code — open it and read the whole relevant path, not just the line the error message names. A symptom two layers deep usually has its trigger further up the call chain, not at the line that finally threw.
2. **Check what changed in that code path**, not just "was there a deploy" — `git log` / `git diff` on the specific file(s) the evidence points to, cross-referenced against the timeline from the step above.
3. **Form exactly one hypothesis**, stated plainly: "I think `<X>` is the root cause, because `<evidence>` shows `<Y>`." Not a list of three plausible guesses — one, falsifiable, tied to a specific piece of evidence gathered above.

## Verify the Hypothesis Against the Evidence

The proof (or disproof) comes entirely from the data the investigation already pulled — the logs, traces, and metrics gathered in the paths above. Not a new test, not an experiment against local or production environments, not synthetic input: the same real data already sitting in front of you, read against the same real code.

1. **Walk the code path by reading it, using the exact conditions the evidence showed** (the input, the record state, the timing) — does this logic, given those conditions, actually produce the observed symptom? If you have to imagine an input the evidence didn't show to make it fail, the hypothesis isn't confirmed yet.
2. **Check the timing lines up** — the suspected code change (commit/deploy) happened *before* the incident window started, not after or so long before that something else is more likely.
3. **Check the error shape matches** — does the exception type, the log message, the failed assertion in the code actually match what the observability evidence recorded? A hypothesis that requires explaining away a mismatch is wrong, not almost-right.
4. **If it doesn't hold up, go back to Root Cause step 1** with what this check revealed — don't force the hypothesis to fit by ignoring the part of the evidence that disagrees with it.
5. **State the confirmed root cause with its evidence trail** — the specific log lines/trace IDs, the specific file and lines of code, the specific commit if one's implicated. This is the primary deliverable (optionally followed by Five Whys, below). Implementing and testing the fix is a separate step, outside this skill, using whatever process the codebase normally uses.

## Push Past the Code Cause (Five Whys)

Once the root cause is confirmed, ask why one layer further: not "what line caused this" but "what let this reach production at all" — a missing test category, a review process gap, an alert that should have fired earlier and didn't. Toyota's Five Whys technique (Taiichi Ohno) treats this as the productive part of the exercise, not an afterthought — going to the *gemba*, the actual place the failure happened, rather than guessing from a meeting room. See `references/methodology.md` #5.

## Red Flags — Stop and Gather Evidence

If you catch yourself thinking:

- "It's obviously the deploy, let's just say that's it" (without checking the diff or the error signature)
- "I already have a dashboard for this" (checking the one dashboard you have instead of asking the system a new question)
- "The error message says enough, no need for the trace/stack"
- "I'll write up the timeline after — I remember what happened"
- "The hypothesis sounds right, no need to check it against the actual code"
- "Let's bump the resource limit and see if it helps" (treating a symptom as if it were the cause)
- "The rollback fixed it, so that proves the deploy was the cause" (consistent with it, not proof of it)
- "Close enough" when the error shape or timing doesn't quite match the hypothesis

**All of these mean: STOP. Go back to gathering evidence, or to verifying the hypothesis against it.**

## Common Rationalizations

| Excuse | Reality |
|---|---|
| "It's obviously the deploy" | Obvious ≠ verified — confirm the error signature actually matches the deploy's diff |
| "No time to write the timeline" | An untracked investigation gets redone from scratch by the next person who hits this same incident |
| "My dashboard shows nothing wrong" | One dashboard is monitoring, not observability — ask the system a new question instead of trusting the one you already built |
| "The hypothesis is plausible, that's enough" | Plausible ≠ verified — walk the actual code against the actual evidence before calling it root cause |
| "The rollback fixed it, that proves the cause" | Consistent with the deploy being the cause, not proof by itself — still confirm the error signature matches the diff |
| "The timing is close enough" | A few minutes off can mean it's the wrong commit — check it, don't eyeball it |

## Quick Reference

| Phase | Key Activities | Success Criteria |
|---|---|---|
| Intake | Get what the reporter already knows: symptom, since when, scope, what's been tried | Have a starting time window and scope before querying anything |
| Triage | Match symptom to Path A/B/C/D | Correct path chosen without running all four |
| Evidence Gathering | Query metrics/logs/traces via MCP or `curl` | Exact error signature, service, and time window known |
| Cross-Service Check | Follow the trace ID / dependency graph if the service's own code looks clean | Confirmed the cause is in this service, or found which one it's actually in |
| Reconstruct Timeline | Log queries and findings as you go; check deploy correlation with the platform's own tool | Timeline written before the hypothesis is formed, not after |
| Root Cause | Trace evidence to code, check the diff, form exactly one hypothesis | Hypothesis stated with the specific evidence behind it |
| Verify Hypothesis | Cross-check the hypothesis against the code and the evidence, by reading — no execution | Error shape and timing both confirmed to match, or hypothesis revised |
| Five Whys | Push past the code cause to the process cause | Answer names a process gap, not just a line of code |

## Common Causes & Solutions

| Symptom | Confirm with evidence | Don't jump straight to |
|---|---|---|
| Error spike right after a deploy | Error signature matches the deploy's diff; timing lines up | Assuming correlation = causation without checking either |
| Latency spike, DB-adjacent | Slow span points at a specific query/lock, not just "the DB" | Concluding "the DB is slow" without the specific query |
| OOM kill | Memory climbs steadily (leak) vs. jumps once (one bad request) — different root causes | Assuming it's a leak without checking the shape of the metric |
| Error rate normal, but users still report issues | Check a narrower segment (region, plan tier, feature flag) — aggregate signals can hide a localized problem | Concluding "nothing's wrong" from an aggregate dashboard |

## Reference

- **references/tool-reference.md** — tool names seen per platform as of 2026 (confirm against `list_tools` before relying on them — see "Discover before using" above) + `curl` fallback for API-key-only access
- **references/methodology.md** — sourcing for the underlying method: Google SRE Book, Charity Majors, PagerDuty, Toyota's Five Whys, John Allspaw/Etsy, Google's Dapper paper, W3C Trace Context, Leslie Lamport