Incident Investigation

Incident Investigation

Overview

This is a purely investigative skill: gather evidence from observability tooling, cross-reference it against the actual codebase, and arrive at a confirmed root cause backed by that evidence. It does not write tests, run code, implement a fix, or deploy anything — the deliverable is a root cause report with its evidence trail, handed off to whatever fixes code next.

Core principle: monitoring answers questions you already knew to ask (known-unknowns); observability lets you ask a question you didn't pre-build an alert for (unknown-unknowns). Most real incidents worth investigating are unknown-unknowns — reaching for the one dashboard you already have is monitoring behavior applied to a problem that needs observability.

The Iron Law

NO ROOT CAUSE CLAIM WITHOUT VERIFICATION AGAINST REAL EVIDENCE

A hypothesis is not a conclusion. Before stating a root cause, cross-check it against the observability evidence and the actual code — both have to agree, not just sound plausible together.

Requires

Works with an MCP server for Grafana, Sentry, Datadog, or New Relic (all ship official servers as of 2026) — or with just an API key/token for any of these, no MCP needed. If you have neither an MCP connection nor an API key for the platform in question, say that explicitly rather than guessing what a dashboard "probably" shows.

Discover before using. Before calling any platform tool by name, check what's actually connected and available in this session — call the MCP server's own tool listing (list_tools or equivalent) rather than assuming. Tool names in this skill and in references/tool-reference.md are examples of the kind of capability to look for (query a time-series metric, search issues by error signature, correlate a deploy against an incident) — verified as of 2026, but these are actively evolving vendor products; the exact name may have changed since. Match by what the tool does, confirmed against what's actually available right now, not by pattern-matching the name written here.

Intake — Understand the Report First

Before querying anything, get what the reporter already knows — don't start blind when a human already has half the evidence. Ask (or extract from what was already said):

• What was observed, in their words — an error message, a screenshot, a customer complaint, an alert notification
• Since when — an exact time if they have one, otherwise "since roughly when" (this seeds the incident window for every path below)
• Scope — one user, one customer, one region, or everyone? A specific endpoint/service, or "the whole app"?
• What's already been tried — a restart, a rollback, a config change? If someone already acted, that action itself is now part of the timeline, not a clean baseline
• Any evidence already in hand — a log line, a trace ID, a request ID, a stack trace pasted into chat. Use it directly instead of re-deriving it from scratch.

A vague report ("it's broken") is still enough to start — go to Path D. Don't block on a perfect report; the paths below are exactly how you fill in what's missing. But don't skip asking, either — a report that already names the service and the approximate time turns a long investigation into a short one.

Triage by Primary Symptom

Pick the path that matches what was reported. Don't run every path — the symptom tells you where to start.

Path A — Error Rate Spike

1. Confirm the window and blast radius.
   • Grafana: query_prometheus with rate(http_requests_total{status=~"5.."}[5m]) — or curl /api/ds/query (see tool-reference.md)
   • Sentry: search_issues filtered to the suspected window — or curl .../issues/?statsPeriod=24h
   • Datadog: search_datadog_error_tracking_issues — or curl /api/v2/logs/events/search with status:error
   • New Relic: list_recent_issues or execute_nrql_query with SELECT count(*) FROM TransactionError SINCE ...
2. Find the error pattern, not just the count. Grafana's Sift tool find_error_pattern_logs groups recurring error signatures automatically — built specifically for this (Grafana Cloud; not confirmed on self-hosted, see tool-reference.md). Sentry: get_event_stacktrace and get_issue_breadcrumbs for the full picture, or analyze_issue_with_seer to let Sentry's own AI triage it first. Datadog: analyze_datadog_logs. New Relic: analyze_entity_logs or list_entity_error_groups.
3. Continue to Reconstruct the Timeline below once you have: the error signature, the exact time it started, and which service/endpoint it's on.

Path B — Latency / Slow Endpoint

1. Confirm which service, not just "it's slow." RED per service (rate, errors, duration) narrows to the offending service before you look inside it.
2. Find the slow spans. Grafana Sift: find_slow_requests. Datadog: apm_latency_bottleneck_summary or apm_query_trace. Sentry: get_trace_details and get_span_details. New Relic: analyze_transactions or analyze_golden_metrics.
3. Check the resource behind it (Path C, step 1) — a slow endpoint is often a starved resource wearing a latency costume.
4. Continue to Reconstruct the Timeline once you have: the specific slow span (DB call, external API, lock wait) and when it started being slow, not just that it currently is.

Path C — Resource Exhaustion (OOM, Restarts, CPU)

1. Pull the USE signals for the resource (utilization, saturation, errors) — CPU/memory over the suspected window. Grafana: query_prometheus for the container/pod metrics. Datadog: get_datadog_metric. New Relic: analyze_golden_metrics.
2. Correlate with restart/kill events, not just the metric shape — get_annotations (Grafana), list_change_events (New Relic), or the platform's own event log often marks the OOM kill or restart directly.
3. Continue to Reconstruct the Timeline once you have: the resource that exhausted, the exact time, and whether it's a leak (climbs steadily) or a spike (sudden jump tied to one request/deploy).

Path D — No Specific Symptom Yet

1. Sweep the Four Golden Signals (latency, traffic, errors, saturation) across the suspected time range first — this tells you which of Paths A/B/C actually applies before you commit to one.
2. Once one signal stands out, jump to the matching path above.

When the Symptom and the Cause Are in Different Services

Any path above can dead-end at "this service is slow/erroring, but its own code looks fine" — that's the signal to check whether the real cause is downstream, in a service this one calls.

1. Follow the trace ID across the service boundary, not just within one service. A distributed trace is a tree: a shared trace ID spans every service touched by one request, and each operation is a span with a parent-child link to the span that called it (Grafana Tempo, Sentry's get_trace_details/get_span_details, Datadog's apm_query_trace, and New Relic's distributed tracing all expose this). The parent-child chain across service boundaries is what tells you where in the call chain the slowness or error actually started, instead of just confirming it was visible where you first looked.
2. Use the platform's dependency-graph tool if one is connected, rather than manually chasing calls service by service — Datadog's search_datadog_service_dependencies and New Relic's list_related_entities exist specifically for this.
3. Propagation only works if every hop actually forwards the trace context (commonly the traceparent HTTP header, per the W3C Trace Context standard) — if a service in the chain doesn't propagate it, the trace breaks there, and that break point is itself useful evidence: it's either where instrumentation is missing, or a real place to keep looking manually. See references/methodology.md #7.

Reconstruct the Timeline

Applies after any path above, before moving to root cause:

1. Write it down as you go — what you queried, what it showed, when. Not from memory afterward.
2. If a deploy/annotation lines up with the incident start, that's a lead, not proof. New Relic's analyze_deployment_impact and Sentry's find_releases correlate this directly instead of you eyeballing two timestamps — use them before concluding a deploy caused it.
3. If timestamps from two different tools don't quite agree, that's expected, not necessarily a red flag by itself — independent systems don't share a clock, and logs/metrics/traces are typically ingested with different delay. Prefer a shared trace ID to line events up across tools when one is available, rather than trying to reconcile wall-clock timestamps down to the second. See references/methodology.md #8.
4. Report what you actually observed, including your own missteps in the investigation so far — an edited timeline is a false one, and the next person investigating a similar incident reads this one first.

Root Cause: From Evidence to Code

The observability evidence above tells you that something broke and roughly where. This section finds why, by reading — not running — the code.

1. Trace the evidence to an actual code path. The stack trace, span, or log line points at specific code — open it and read the whole relevant path, not just the line the error message names. A symptom two layers deep usually has its trigger further up the call chain, not at the line that finally threw.
2. Check what changed in that code path, not just "was there a deploy" — git log / git diff on the specific file(s) the evidence points to, cross-referenced against the timeline from the step above.
3. Form exactly one hypothesis, stated plainly: "I think <X> is the root cause, because <evidence> shows <Y>." Not a list of three plausible guesses — one, falsifiable, tied to a specific piece of evidence gathered above.

Verify the Hypothesis Against the Evidence

The proof (or disproof) comes entirely from the data the investigation already pulled — the logs, traces, and metrics gathered in the paths above. Not a new test, not an experiment against local or production environments, not synthetic input: the same real data already sitting in front of you, read against the same real code.

1. Walk the code path by reading it, using the exact conditions the evidence showed (the input, the record state, the timing) — does this logic, given those conditions, actually produce the observed symptom? If you have to imagine an input the evidence didn't show to make it fail, the hypothesis isn't confirmed yet.
2. Check the timing lines up — the suspected code change (commit/deploy) happened before the incident window started, not after or so long before that something else is more likely.
3. Check the error shape matches — does the exception type, the log message, the failed assertion in the code actually match what the observability evidence recorded? A hypothesis that requires explaining away a mismatch is wrong, not almost-right.
4. If it doesn't hold up, go back to Root Cause step 1 with what this check revealed — don't force the hypothesis to fit by ignoring the part of the evidence that disagrees with it.
5. State the confirmed root cause with its evidence trail — the specific log lines/trace IDs, the specific file and lines of code, the specific commit if one's implicated. This is the primary deliverable (optionally followed by Five Whys, below). Implementing and testing the fix is a separate step, outside this skill, using whatever process the codebase normally uses.

Push Past the Code Cause (Five Whys)

Once the root cause is confirmed, ask why one layer further: not "what line caused this" but "what let this reach production at all" — a missing test category, a review process gap, an alert that should have fired earlier and didn't. Toyota's Five Whys technique (Taiichi Ohno) treats this as the productive part of the exercise, not an afterthought — going to the gemba, the actual place the failure happened, rather than guessing from a meeting room. See references/methodology.md #5.

Red Flags — Stop and Gather Evidence

If you catch yourself thinking:

• "It's obviously the deploy, let's just say that's it" (without checking the diff or the error signature)
• "I already have a dashboard for this" (checking the one dashboard you have instead of asking the system a new question)
• "The error message says enough, no need for the trace/stack"
• "I'll write up the timeline after — I remember what happened"
• "The hypothesis sounds right, no need to check it against the actual code"
• "Let's bump the resource limit and see if it helps" (treating a symptom as if it were the cause)
• "The rollback fixed it, so that proves the deploy was the cause" (consistent with it, not proof of it)
• "Close enough" when the error shape or timing doesn't quite match the hypothesis

All of these mean: STOP. Go back to gathering evidence, or to verifying the hypothesis against it.

Common Rationalizations

Quick Reference

Common Causes & Solutions

Reference

• references/tool-reference.md — tool names seen per platform as of 2026 (confirm against list_tools before relying on them — see "Discover before using" above) + curl fallback for API-key-only access
• references/methodology.md — sourcing for the underlying method: Google SRE Book, Charity Majors, PagerDuty, Toyota's Five Whys, John Allspaw/Etsy, Google's Dapper paper, W3C Trace Context, Leslie Lamport