Splunk Log Investigation

Splunk Log Investigation

Overview

Investigation is funnel, then trace: narrow from "something is wrong" to the exact failing events, then follow one bad transaction end-to-end across services. The discipline that separates fast investigations from flailing: establish the timeline before forming theories, and confirm root cause with evidence before declaring it.

Core principle: find the change. Most incidents are a deviation from a known-good baseline — a deploy, a config push, a traffic shift, a dependency failure. Your job is to locate the deviation in time and in the data.

Investigation Workflow

1. SCOPE      What, where, when? Pin the blast radius and a precise time window.
2. BASELINE   Compare now vs a healthy period. What changed?
3. ISOLATE    Narrow to the failing slice (host/service/endpoint/user cohort).
4. TRACE      Follow one failing request/session across all relevant sources.
5. ROOT CAUSE Confirm the cause with evidence — not the first correlation.
6. TIMELINE   Assemble the sequence; hand off / document.

Do not skip to step 5. The most common failure mode is locking onto the first suspicious log line and rationalizing it as the cause.

1. Scope — pin the window

index=* (error OR fail* OR exception OR 5*) earliest=-2h
| timechart span=1m count by index

Find when the spike starts. Then snap your window tightly around it (earliest=-1h@m latest=now). A precise window makes every later search faster and clearer.

2. Baseline — what changed?

Compare the incident window against a healthy one. timewrap is built for this:

index=web sourcetype=access_combined status>=500
| timechart span=5m count
| timewrap 1d                      ← overlay yesterday same time

Also check for deploys/config changes in the same window — these are the usual culprits:

index=ci OR index=deploy earliest=-3h
| table _time, service, version, action, user
| sort _time

3. Isolate — narrow the failing slice

Break the error down by every dimension until one stands out:

index=web status>=500 earliest=-1h@m
| stats count by host, uri_path, status
| sort - count

If host=web-07 carries 90% of errors → host problem. If one uri_path dominates → code/dependency problem. If errors are spread evenly → upstream/shared dependency. The distribution of failures tells you what kind of problem it is.

4. Trace — follow one transaction

Pick one failing identifier (request_id, trace_id, session_id, order_id) and follow it across every source. This is where you see the actual failure, not just its symptom:

index=* (request_id="abc-123")
| sort _time
| table _time, index, sourcetype, host, level, message

When there's no shared ID, correlate by time + key with stats (not join):

index=app OR index=db earliest=-15m
| stats values(error) as errors, values(query_ms) as latencies,
        earliest(_time) as start, latest(_time) as end by session_id
| where isnotnull(errors)

For latency regressions, look at the distribution, not the average — averages hide tail latency:

index=app sourcetype=apptrace earliest=-1h
| stats perc50(latency_ms) as p50, perc95(latency_ms) as p95,
        perc99(latency_ms) as p99, count by endpoint
| sort - p99

5. Root cause — confirm, don't assume

Before declaring a cause, it must explain:

• Timing — the cause appears at/just before the symptom onset.
• Scope — the cause covers the same blast radius (same hosts/endpoints/users).
• Mechanism — there's a plausible causal chain, ideally visible in the trace.

If a candidate cause fails any of these, keep looking. Correlation in one host ≠ cause if the symptom spans all hosts.

6. Timeline — assemble and hand off

index=* (request_id="abc-123" OR host="web-07" OR action="deploy")
  earliest=-2h@m
| sort _time
| table _time, source, host, action, level, message

Produce: onset time, trigger/change, mechanism, blast radius, and the evidence search for each claim.

Quick Reference — investigation searches

Common Mistakes

REQUIRED COMPANION: Use splunk-spl-authoring for query construction and splunk-spl-optimization when investigation searches are too slow over large windows. For a structured, root-cause-first debugging mindset that applies beyond Splunk, see superpowers:systematic-debugging.