Build an incident timeline from timestamps
Postmortems go wrong when the story is written before the facts are ordered. Build the timeline first: every event with its source and a UTC timestamp, sorted. The narrative then falls out of the gaps.
Procedure
Fix the timezone first. Convert everything to UTC and record the source clock for each line. A timeline mixing local pod time with UTC chat time is worse than no timeline.
Pull machine events into one tab-separated file, one event per line,
ts<TAB>source<TAB>event: kubectl get events --sort-by=.lastTimestamp
-o custom-columns=TIME:.lastTimestamp,REASON:.reason,MSG:.message > k8s-events.tsv git log --since="6 hours ago" --pretty='%cI%x09git%x09%h %s' >> timeline.tsvAdd human events from chat and ticket exports with their own timestamps: jq -r '.[] | "(.ts)\tslack\t(.user): (.text)"' incident-export.json >> timeline.tsv
Include observability markers: the first alert fired and the metric-deviation onset. The onset usually precedes the alert by minutes, and that gap is itself a finding: curl -s 'localhost:9090/api/v1/query?query=rate(err[1m])>0.02&time='
Sort, dedupe, and clamp to the incident window ± 30 min: sort -k1,1 timeline.tsv | awk '!seen[$0]++' > timeline-sorted.tsv
Mark two anchors explicitly:
T0is the first customer impact (not the first alert);Tresolvedis when impact ended (metric back under threshold, not "we stopped looking"). Derive MTTD = T0→alert, MTTM = alert→mitigation, MTTR = T0→Tresolved.Leave gaps visible. "07:14 deploy, 07:31 alert" is a 17-minute blind spot worth naming; filling it with prose hides a missing detection.
Pitfalls
- Pulling timestamps from unsynchronised clocks without noting skew — 30 s of node skew reorders events and invents causality.
- Using the deploy time as the cause without evidence. Mark causal claims as hypotheses on separate lines from the fact lines.
- Taking T0 as the alert time, which understates customer impact and flatters the SLO.
- Mixing local browser or chat display time with UTC server time in a single column.
Verification
wc -l timeline-sorted.tsv # sum of sources minus dupes
awk -F'\t' '{print $1}' timeline-sorted.tsv | sort -c # sorted ascending
grep -c 'T0\|Tresolved' timeline-sorted.tsv # both anchors present
Report: event count by source, the two anchors with MTTD/MTTM/MTTR, and the single largest unexplained gap named as a detection finding.