Tempo

Tempo — Distributed Tracing for the Grafana Stack

Overview

Grafana Tempo is a high-scale, cost-efficient distributed tracing backend. Unlike Jaeger or Zipkin, Tempo stores traces in object storage (S3, GCS, Azure Blob) with no additional dependencies — it requires no Elasticsearch or Cassandra. Traces are queried via TraceQL, a query language similar to LogQL, and visualized in Grafana with native trace-to-logs and trace-to-metrics correlations. Tempo accepts OTLP, Jaeger, Zipkin, and Zipkin Protobuf formats, making it a drop-in replacement for most existing tracing backends.

When to Use

• Storing distributed traces generated by OpenTelemetry-instrumented services
• Replacing Jaeger/Zipkin with a cheaper, object-storage-backed backend
• Correlating traces with Loki logs and Prometheus metrics in Grafana
• Querying spans by attribute, duration, or service name via TraceQL
• High-volume tracing where Jaeger's Elasticsearch costs are prohibitive

Installation

# Install with Helm (single-binary for small clusters)
helm repo add grafana https://grafana.github.io/helm-charts
helm install tempo grafana/tempo \
  --namespace monitoring \
  --create-namespace \
  --set tempo.storage.trace.backend=local

# For production: distributed mode with S3
helm install tempo grafana/tempo-distributed \
  --namespace monitoring \
  --values tempo-values.yaml

# Verify
kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo

Key Patterns

Tempo Configuration — S3 Backend

# tempo-values.yaml (for tempo-distributed chart)
storage:
  trace:
    backend: s3
    s3:
      bucket: my-tempo-traces
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      # Uses IRSA on EKS or explicit credentials:
      # access_key: ...
      # secret_key: ...
    pool:
      max_workers: 100
      queue_depth: 10000

# Accept multiple trace formats
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"
  jaeger:
    protocols:
      thrift_http:
        endpoint: "0.0.0.0:14268"
      grpc:
        endpoint: "0.0.0.0:14250"
  zipkin:
    endpoint: "0.0.0.0:9411"

# Retention
compactor:
  compaction:
    block_retention: 336h  # 14 days

# Search
search_enabled: true

OpenTelemetry Collector — Sending Traces to Tempo

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  # Sample 10% of traces (reduce cost for high-volume services)
  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 10

exporters:
  otlp:
    endpoint: "tempo.monitoring.svc.cluster.local:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, probabilistic_sampler]
      exporters: [otlp]

Python Service — OpenTelemetry Instrumentation

# requirements: opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

resource = Resource.create({"service.name": "my-api", "service.version": "1.2.0"})
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# Auto-instrument FastAPI (adds spans for every request)
FastAPIInstrumentor.instrument_app(app)

# Manual span for critical operations
tracer = trace.get_tracer("my-api")

def process_order(order_id: str):
    with tracer.start_as_current_span("process-order") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.source", "web")
        try:
            result = db.execute_order(order_id)
            span.set_attribute("order.status", "success")
            return result
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.StatusCode.ERROR, str(e))
            raise

TraceQL — Querying Traces

# Find all traces from the checkout service
{ .service.name = "checkout" }

# Find traces with errors
{ status = error }

# Find slow spans (>500ms) in the payment service
{ .service.name = "payment" && duration > 500ms }

# Find traces that hit the database and were slow
{ span.db.system = "postgresql" && duration > 200ms }

# Find traces with a specific HTTP path
{ span.http.route = "/api/checkout" && span.http.status_code >= 500 }

# Aggregate: p99 latency by service
{ } | rate() by (.service.name)

Grafana — Tempo Data Source Configuration

# Grafana datasource provisioning
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    url: http://tempo.monitoring.svc.cluster.local:3100
    jsonData:
      # Correlate with Loki logs
      tracesToLogs:
        datasourceUid: loki
        tags: ["job", "instance", "pod", "namespace"]
        filterByTraceID: true
        filterBySpanID: true
      # Correlate with Prometheus metrics
      tracesToMetrics:
        datasourceUid: prometheus
        tags: [{key: "service.name", value: "job"}]
        queries:
          - name: "Request rate"
            query: "rate(http_requests_total{$$__tags}[5m])"
      # Enable Service Graph view
      serviceMap:
        datasourceUid: prometheus
      # Enable search
      search:
        hide: false
      nodeGraph:
        enabled: true

Service Graph Metrics (Prometheus)

# Tempo generates service graph metrics automatically when configured:
# tempo-values.yaml
metricsGenerator:
  enabled: true
  config:
    storage:
      remote_write:
        - url: http://prometheus.monitoring.svc.cluster.local:9090/api/v1/write

# This produces Prometheus metrics like:
# traces_service_graph_request_total
# traces_service_graph_request_failed_total
# traces_service_graph_request_duration_seconds_bucket
# Which power the Grafana Service Graph panel

Common Commands

# Check Tempo health
curl http://tempo:3100/ready

# List available tags for TraceQL
curl http://tempo:3100/api/search/tags

# Search traces via API
curl "http://tempo:3100/api/search?tags=service.name%3Dmy-api&limit=20"

# Fetch a specific trace by ID
curl "http://tempo:3100/api/traces/<trace-id>"

# Check Tempo metrics
curl http://tempo:3100/metrics | grep tempo_ingester

# Port-forward for Tempo UI
kubectl port-forward -n monitoring svc/tempo 3100

Pitfalls

• Trace retention vs storage cost: Tempo stores everything by default; always set block_retention in the compactor config — 14 days is a reasonable default for most teams
• Head-based vs tail-based sampling: configure sampling at the collector level, not in individual services; tail-based sampling (keep only slow/error traces) requires the tail_sampling processor in the OpenTelemetry Collector
• TraceQL search requires the search component: in distributed mode, search only works when the query-frontend and querier components are running — a common oversight when deploying minimal setups
• OTLP gRPC vs HTTP: gRPC (port 4317) is preferred for high throughput; HTTP (4318) is easier to debug but has higher overhead
• Service Graph requires metrics generator: the Service Graph Grafana panel only works if you enable Tempo's metrics generator and remote-write to Prometheus — check this before promising the feature

Related Skills

• loki — log aggregation (correlate with traces)
• prometheus-recording-rules — metrics alongside tracing
• alloy — OpenTelemetry collector that feeds Tempo
• observability-engineer — full Grafana stack strategy
• opentelemetry-instrumentation — instrumenting services for tracing

GitNexus Index

Index path: /Users/localuser/.claude/skills/tempo/.gitnexus Created: 2026-05-24