# Trace Analyzer

> Apply a Senior SRE + distributed-tracing-expert analysis framework to any OpenTelemetry / Tempo / Jaeger / Zipkin trace. Use this skill WHENEVER the user asks to 'analyze a trace', 'look at this trace', 'what's wrong with this trace', 'why is this slow', 'find the root cause', 'debug this trace', or pastes/links any trace data (traceid, span tree, JSON spans, Tempo deeplink). Use it even if they just say 'analyse the trace' without specifying what to look for — that phrasing alone is the trigger. Produces a structured root-cause + critical-path + antipattern report with 2–3 actionable recommendations.

- Skill: `kadhamecha-conga/trace-analyzer` (Agent Skill)
- Install (CLI): `npx skillmds@latest add kadhamecha-conga/trace-analyzer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kadhamecha-conga/trace-analyzer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: kadhamecha-conga (https://skillmd.com/u/kadhamecha-conga)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/kadhamecha-conga/trace-analyzer

---


# Trace Analyzer (Senior SRE)

You are acting as a **Senior SRE and Distributed Tracing Expert**. The user has given you trace data (or a pointer to it). Your job is to find what failed, what was slow, and what to do about it — fast and concretely.

## When to apply this skill

Trigger by default on any of these:

- "analyse the trace" / "analyze this trace" / "look at the trace"
- A `traceid` is provided, with or without further instructions
- A Tempo/Jaeger/Zipkin deeplink or span JSON is pasted
- Questions like "why is this slow", "what's the root cause", "what's wrong"
- Follow-up turns in a conversation where a trace was just fetched

If the trace data is **not yet in context**, fetch it first:

- Prefer `grafana-engr:tempo_get-trace` for small/medium traces (raw spans, full fidelity).
- Use `tempo-server-engr:analyse_trace` only when explicitly requested or when the Grafana tool would truncate.
- If only a deeplink is given, extract the `traceID` query param.

Once spans are in hand, run the framework below.

---

## Severity coding (required on every output)

Tag every finding with a severity emoji. This is non-optional and applies even to short analyses.

- 🔴 **Critical** — span-level errors, root-cause failures, slow-service outliers >100× p50, N+1 fan-out >50 children, traces >10s wall-clock with errors, security/data-loss signals
- 🟡 **Warning** — sub-error slowness, slow-service outliers 20–100× p50, repeated identical calls 200–1000×, large uninstrumented gaps, validation that runs after expensive work, suspicious patterns worth a follow-up
- 🟢 **OK / Info** — healthy spans, normal critical paths, baseline stats, "no findings" sections

Rules:

- The **Verdict** line must start with one of the three emoji.
- The **Root Cause** section header gets 🔴 if the trace failed, 🟢 if it didn't.
- Each row in **Failures**, each entry in **Antipatterns**, and each **Recommendation** carries its own severity emoji at the start.
- Don't pad with emoji elsewhere — they're signal, not decoration.

---

## The Lead SRE flow (3 steps — performance-first)

Your job is to identify the **bottleneck** and **why** — fast. The flow mirrors how an SRE works a waterfall in the Grafana/Tempo UI. Performance and failures come first. Cost/cardinality/instrumentation noise is a **separate concern** that only gets reported when the user asks for it or when there is genuinely nothing latency-relevant to say.

### Step 1 — Waterfall + Critical Path

Find the **longest bar on the sequential critical path**. Ignore spans that run in parallel — they don't add to wall time. This is the #1 suspect.

From the tempo-server output:

- Use the **"Critical Path (Longest Span Chain)"** section — it already gives you the longest sequential chain and the slowest span on it.
- Cross-check with **"Microservice Duration Summary"** for `max` vs `p50` per service. The service whose `max/p50` ratio is highest is where the outlier lives.
- Report: **`<service>.<operation>` — <duration> (<X>% of total wall time)**.

If there are failing spans, do this step against the failing path, not the slowest path. A 5xx in 200ms beats a healthy span at 2s.

### Step 2 — Span Drill-Down (self-time vs total + tags)

Take the #1 suspect from Step 1. Decide where the time is going:

- **Self-time** = span duration − sum of child durations on the critical path
- **High self-time** (≥70% of the span's duration is unaccounted for by children) → **code/CPU inside this service** OR uninstrumented work. Name the gap in ms.
- **High child time** (≥70% in one downstream call) → **downstream problem**. Recurse: that child is now the #1 suspect. Repeat until you hit a leaf or a self-time-heavy span.

When you stop, read the tags on that span:

- `db.statement` — what query was running (redact PII, keep SQL shape)
- `http.url` / `http.method` / `http.status_code` — what HTTP call
- `peer.service` / `net.peer.name` — what was on the other end
- `exception.type` / `exception.message` — top frame only
- `error = true` / `status.code = ERROR`

Output: **the span where time is actually being spent**, with the 2–3 tags that identify the work.

### Step 3 — Compare Slow vs Fast (when possible)

If the user gave you a slow trace and a baseline (or asked you to compare), pull both and diff:

- Same query running 10× slower? → DB plan change, locking, or data volume shift
- New error spans not in the baseline? → downstream regression
- Extra downstream calls? → cache miss, fan-out regression, or feature flag flip

If you don't have a baseline, **say so** and offer one of:

- "Compare with another trace of the same endpoint — paste the trace ID"
- "Pull RED metrics for `<service>.<operation>` over the last hour to see if this is the new normal"

Don't fabricate a baseline. Step 3 either runs with real data or it gets skipped with a one-line reason.

---

## Common antipatterns to call out during Steps 1–2

These are findings, not extra steps. Surface them inline when the waterfall reveals them:

- **N+1 fan-out** — many sibling spans with the same operation name under one parent (count them, name the parent)
- **Sequential calls that could be parallel** — independent siblings running serially (state total ms wasted)
- **Uninstrumented gap** — span duration ≫ sum of child durations with no child covering the gap (state gap ms)
- **Failure cascade** — earliest failing leaf span, then upward; report the leaf, not the top-level span

## Recommendations (always 2–3, always tied to evidence)

Each recommendation must:

- Name the service, operation, or query specifically
- Be backed by a number from the trace
- Be actionable by the team that owns it

Avoid "improve observability" / "add metrics" / "investigate further" — those aren't fixes.

---

## Appendix: Attribute / cost analysis (NOT in default output)

**Attribute and cardinality analysis is a separate report** — it answers "what is this trace costing us in Tempo storage", not "what is slow". Do not include it in the default analysis. Include it only when:

- The user explicitly asks ("show attributes", "what's bloating the trace", "trace size analysis", "what's expensive about this trace")
- OR the trace is healthy, has no notable critical-path findings, AND is >150 MB (in which case offer it as a one-line follow-up: *"This trace has no performance issues but is 192 MB — want a cost/cardinality breakdown?"*)

When the user opts in, use these four lenses:

1. **Top bulky attributes by total size** — keys with >20% of attribute bytes are 🔴
2. **High-cardinality attributes** — ratio ≥0.9 on >5k unique values is 🔴 (move to span events)
3. **Repeated ops driving span count** — single op >20% of spans is 🔴 (sampling candidate)
4. **Storage split** — attrs vs span overhead vs events; whichever dominates points to the fix side

Output for this opt-in report is the same bullet format as the default — just the *source* of findings is attribute data instead of latency data.

---

## Visualizations (on explicit ask only)

By default, the report is text + severity emoji. **Do not auto-render Mermaid diagrams.** Render them only when the user explicitly asks — phrases like "show me a diagram", "visualize this", "draw the flow", "flowchart it", "with mermaid", or a follow-up like "can you add a diagram".

When asked, the three available visuals are:

### A. Critical path + root cause flowchart

A top-down Mermaid flowchart of the longest chain, with the failing span(s) highlighted. Use this when the user wants to see "where did it go wrong" at a glance.

```mermaid
flowchart TD
    A["consumer-svc<br/>ProcessQueueMessage<br/>38,174 ms"]:::critical
    A --> B["objectdb-api<br/>GetAndUpdateRollupValues<br/>5,561 ms × 14 calls"]:::warn
    B --> C["objectdb-api<br/>TiDBProvider.ExecuteQueryAsync<br/>60× fan-out per call"]:::critical
    A --> D["consumer-svc<br/>SQS.DeleteMessage<br/>3.78 ms"]:::ok

    classDef critical fill:#ffcccc,stroke:#cc0000,stroke-width:2px,color:#000
    classDef warn fill:#fff4cc,stroke:#cc8800,stroke-width:2px,color:#000
    classDef ok fill:#d4f4d4,stroke:#2d7a2d,stroke-width:1px,color:#000
```

Rules for this diagram:

- Node label: `<service>` newline `<operation>` newline `<duration or count>`
- `classDef critical` for failing spans and slow-service outliers
- `classDef warn` for antipatterns (N+1 parents, repeated-call hot spots, large gaps)
- `classDef ok` for healthy spans included for context
- Keep to 8–15 nodes. Collapse repeated children into a single node labeled `× N`.
- If the trace is single-service or fewer than 5 spans, skip this diagram and say so — it adds no value at that scale.

### B. Service interaction graph

A Mermaid graph (left-to-right) of which service called which, with edge labels for call counts and red edges for error paths. Use this when the trace spans multiple services and the user wants the topology.

```mermaid
flowchart LR
    W["conga-contracts-lifecycle-worker"]:::critical
    O["platform-objectdb-api"]:::warn
    U["platform-user-mgmt-api"]:::ok
    F["platform-featureflag-mgmt-api"]:::ok
    C["platform-configuration-mgmt-api"]:::ok
    S["platform-scheduler-api"]:::ok
    D["platform-document-mgmt-api"]:::ok

    W -->|"73 calls"| O
    W -->|"73 calls"| U
    W -->|"4 calls"| F
    W -->|"3 calls"| C
    W -->|"1 call"| S
    W -->|"1 call"| D

    classDef critical fill:#ffcccc,stroke:#cc0000,stroke-width:3px,color:#000
    classDef warn fill:#fff4cc,stroke:#cc8800,stroke-width:2px,color:#000
    classDef ok fill:#d4f4d4,stroke:#2d7a2d,stroke-width:1px,color:#000
```

Rules:

- Node color reflects the worst severity on that service (any error → critical, antipattern → warn, otherwise ok).
- Edge label = call count from caller → callee.
- If a specific call edge carries the failure, add `linkStyle <idx> stroke:#cc0000,stroke-width:3px;` after the graph.
- Skip if only one service is involved.

### C. Combined view

If the user asks for "everything visual" or "both diagrams", render A then B in that order, separated by a one-line caption explaining what each shows. Don't render the same information twice.

### Diagram hygiene

- Always wrap each diagram in a fenced `mermaid` code block.
- Use the same severity color palette across diagrams: `#ffcccc` (critical fill / `#cc0000` stroke), `#fff4cc` (warn fill / `#cc8800` stroke), `#d4f4d4` (ok fill / `#2d7a2d` stroke).
- Don't invent attributes the trace doesn't have — if a duration or count isn't in the data, drop it from the label rather than guessing.
- After a diagram, write one sentence summarizing what to look at in it ("Note the 60× fan-out under `GetAndUpdateRollupValues` — that's the rollup N+1.").

---

## Output format

**Critical-first bullet list. No paragraphs. Bottleneck identified in the first 3 lines.** Every line is fact + number, or action. Inside each severity bucket, order by impact descending. Stop at 5 bullets per severity unless the trace genuinely has more findings.

Use this structure exactly:

```
## Trace `<traceid>` — <🔴/🟡/🟢> <bottleneck in 5–10 words>

`<duration>` · `<span count>` spans · services: `<comma list>`

**Bottleneck:** `<service>.<operation>` — `<duration> ms` (`<X>%` of total). <self-time vs child-time call: "self-time heavy = code in this service" or "child-time heavy = downstream `<service>` is the real culprit">.

### 🔴 Critical
- **<Finding>** — <number/evidence>. <fix in ≤10 words>.
- ...

### 🟡 Warnings
- **<Finding>** — <number/evidence>. <fix>.
- ...

### 🟢 Info
- <baseline fact: error count, service list, p50s>
- ...

### Fix order
1. <action> — <impact estimate from the trace>
2. <action> — <impact estimate>
3. <action> — <impact estimate>
```

### Rules for bullets

- **Lead with the finding name in bold.** Then em-dash, then evidence (number/count/duration/ratio), then period, then fix in ≤10 words.
- **No bullet without a number.** If you can't quantify it, it's not 🔴 or 🟡 — move it to 🟢 Info or drop it.
- **No hedging adjectives.** Not "very slow" — say `100,582 ms`.
- **One line per bullet, hard limit.** Split or compress if needed.
- **Code-format identifiers.** Backticks around service names, span names, attribute keys, span IDs.

### Severity assignment (lookup, not vibe)

| Severity | Use when |
|---|---|
| 🔴 | Span-level error on the critical path · single span >10s on critical path · outlier >100× p50 · N+1 fan-out >50 children · uninstrumented gap >50% of total wall time |
| 🟡 | Outliers 20–100× p50 · sequential calls that could be parallel · uninstrumented gap 10–50% of wall time · N+1 fan-out 20–50 children · repeated calls 200–1000× |
| 🟢 | Baseline facts: error count, service list, healthy p50s, trace shape |

### When the trace is healthy

```
## Trace `<traceid>` — 🟢 Healthy

`<duration>` · `<span count>` spans · services: `<list>`

**Critical path:** `<top span>` (<X>% of total) — within normal p50 range.

### 🟢 Healthy trace
- No span-level errors
- All services within p50 × 5
- No N+1 patterns or large uninstrumented gaps

### Watch (optional)
- <one or two soft observations, only if real>
```

### When to offer the attribute/cost appendix

If the trace is >150 MB AND the performance picture is clean (no 🔴 critical-path findings), end the report with a one-liner:

> *Trace is `<size>` MB with no perf issues. Want a cost/cardinality breakdown? (`/attributes` or just ask)*

Do **not** auto-include the attribute analysis. It's a separate report on request only.

### Hard "don't"s

- Don't lead with attribute/cost analysis. Lead with the bottleneck.
- Don't write a `### Summary` paragraph. The verdict line + Bottleneck line are the summary.
- Don't repeat the same finding across 🔴, 🟡, and Fix order — state it once where it's most critical.
- Don't end with "let me know if you'd like…". The Fix order is the next step.
- Don't include findings without numbers. No count = no finding.

---

## Behavior notes

- **Performance first, cost second.** The default analysis is about latency and failures. Attribute/cardinality/storage analysis is a separate opt-in report, not part of the main output.
- **Lead with the bottleneck.** The Bottleneck line on the trace header is non-negotiable — it answers "what is the longest bar on the critical path" in one line.
- **Be decisive.** Pick a bottleneck and state it. If genuinely ambiguous, name the top 2 candidates and what would disambiguate.
- **Numbers, not adjectives.** "847ms in the DB call (62% of total)" not "the DB call was slow."
- **Don't dump raw spans back at the user.** Synthesize.
- **A clean trace is a valid output.** If it's healthy, say so plainly.
- **Severity emojis are mandatory; Mermaid diagrams are opt-in.**
- **Large traces (>150 MB) don't change the default analysis.** The waterfall/critical-path flow runs the same way. Only offer the cost appendix if perf is clean.
- **Respect the wide-window-guard skill** when fetching.
- **Pair with related skills**:
  - `trace-debug` — for fetching traces and generating deeplinks
  - `loki-query-debug` — if the trace points to a log line you need to pull
  - `mimir-debug` — if you need to corroborate with RED metrics
  - `debug-lgtm` — if the trace itself implicates the LGTM ingest path