Distributed Tracing

When to Use

• Use when: a request touches > 1 service and you can't tell where latency is coming from
• Use when: adding observability to a new microservice before it goes to production
• Use when: correlating logs across services using a shared trace ID
• Use when: debugging intermittent slowness that only appears under load
• Do NOT use for: single-process performance profiling — use a profiler (py-spy, clinic.js)
• Do NOT use for: error aggregation alone — traces complement, not replace, error tracking

Core Concepts

Trace   = one request's full journey across all services
Span    = one unit of work within a trace (DB query, HTTP call, function)
Context = trace ID + span ID propagated across process boundaries via headers

W3C traceparent header (de facto standard):
  traceparent: 00-{traceId:32hex}-{spanId:16hex}-{flags:2hex}
  Example:     00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
                  └─ version    └─ trace ID (128-bit)  └─ span ID  └─ sampled

Setup (Node.js — SDK init must run before any imports)

// tracing.js — must be required FIRST: node -r ./tracing.js app.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { Resource } = require('@opentelemetry/resources');
const { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION } = require('@opentelemetry/semantic-conventions');

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]:    'checkout-service',
    [SEMRESATTRS_SERVICE_VERSION]: process.env.APP_VERSION ?? 'unknown',
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()], // auto-instruments HTTP, Express, DB drivers
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown());

Auto-instrumentation covers Express, fetch/http, pg, redis, mongoose — no manual work for common libraries.

Manual Spans

const { trace, context, SpanStatusCode } = require('@opentelemetry/api');
const tracer = trace.getTracer('checkout-service');

async function processPayment(orderId, amount) {
  return tracer.startActiveSpan('payment.process', async (span) => {
    span.setAttributes({
      'order.id':     orderId,
      'payment.amount': amount,
      'payment.currency': 'USD',
    });
    try {
      const result = await paymentProvider.charge(orderId, amount);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.recordException(err);                       // captures stack trace
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();                                      // always end the span
    }
  });
}

Setup (Python)

# At app startup — before other imports
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

provider = TracerProvider(resource=Resource({SERVICE_NAME: "order-service"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

FastAPIInstrumentor.instrument()
SQLAlchemyInstrumentor().instrument(engine=engine)

# Manual span
tracer = trace.get_tracer("order-service")
with tracer.start_as_current_span("inventory.check") as span:
    span.set_attribute("product.id", product_id)
    result = check_inventory(product_id)

Sampling Strategy

Always-on (100%)   → dev/staging only — too expensive in high-traffic prod
Head-based (%)     → decide at trace root; cheap but misses rare errors
                     OTEL_TRACES_SAMPLER=parentbased_traceidratio
                     OTEL_TRACES_SAMPLER_ARG=0.1   ← 10% of traces

Tail-based         → collect all spans, decide after seeing full trace
                     → captures 100% of errors and slow requests
                     → requires a collector (OTel Collector tail sampler)
                     → recommended for production

# OTel Collector tail sampling — keep all errors + slow requests
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors,     type: status_code, status_code: { status_codes: [ERROR] }
      - name: slow,       type: latency,     latency: { threshold_ms: 1000 }
      - name: sample-10,  type: probabilistic, probabilistic: { sampling_percentage: 10 }

Correlating Traces with Logs

// Inject trace ID into every log entry — enables log→trace navigation
const { trace, context } = require('@opentelemetry/api');

function getTraceContext() {
  const span = trace.getActiveSpan();
  if (!span) return {};
  const { traceId, spanId } = span.spanContext();
  return { traceId, spanId };
}

// With Pino
const logger = pino({
  mixin: getTraceContext,   // automatically added to every log line
});

// Log output:
// {"level":30,"msg":"Payment processed","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"00f067aa0ba902b7"}

In Grafana: paste traceId into Tempo → click "Logs for this span" to jump to Loki.

Common Pitfalls

Anti-Fake-Pass Rules

Before claiming tracing is done, you MUST show:

• SDK init runs before all other imports — not mid-file
• service.name resource attribute set — not default "unknown_service"
• Manual spans used for business-critical paths (not just HTTP auto-instrumentation)
• Errors recorded on spans with span.recordException() + SpanStatusCode.ERROR
• span.end() called in all paths — including error branches
• Trace ID injected into log entries for log↔trace correlation
• Sampling strategy defined — not 100% in production

Reference: gates/anti-fake-pass-gate.md