Distributed Tracing

Distributed tracing

In a mesh of a dozen services, "checkout is slow" names a symptom with no address. A log gives you one service's account; a trace gives you the whole conversation with timing attached. Profile each service alone and you polish the fast ones while the time drains into a call three hops down you never saw.

Method

1. Propagate context or the trace snaps at hop one. Verify every service forwards the traceparent header (W3C Trace Context) or B3. A dropped header spawns a fresh root, and the child spans orphan into a tree you can never rejoin to the parent.
2. Name spans for the operation, not the method. handleRequest says nothing; GET /orders/:id and SELECT orders by user say what ran. Set span.kind to server, client, or producer so the waterfall nests parent and child correctly.
3. Follow the critical path, not the sum. In the waterfall, trace the chain of spans a later span actually blocked on. A 400ms span running beside a 900ms sibling is free; the 900ms one owns the budget. Attack the longest bar on the blocking path.
4. Ride baggage for cross-cutting facts. Put user.tier or feature.flag into OpenTelemetry baggage so it propagates to every downstream span. Later you query "traces where tier=free and duration>2s" without threading that value through every signature by hand.
5. Hunt the gap between spans, not only the long ones. Dead time with no child span open is queueing, pool wait, or GC that your instrumentation missed. A 300ms hole before the database span is the connection pool starving, not the query executing.
6. Sort errored traces by where status flipped. Filter status=error and find the deepest span that set it. Parents inherit the failure, so the leaf that first went red is the origin; read its events for the exception.

Signals

• Can you point to the one span on the critical path that owns most of the latency?
• Does a single trace id ride the request from edge to database?
• On a failed request, did you locate the deepest span that first set error?

Boundaries

Tracing shows where time and errors move between services; it will not explain why one function is slow inside its own span. For that, drop to a profiler or add metrics. Sampling means a given one-off trace may not exist at all: tail-based sampling keeps errors, head-based may discard them.