Monitoring Tempo

Grafana Tempo Monitoring Skill

Query and analyze distributed traces using Grafana Tempo and TraceQL.

API Conventions

Authentication

Tempo API uses Basic auth, Bearer token, or tenant header (X-Scope-OrgID) — injected by connection.

Base URL

• Tempo API: http://<host>:3200/
• Use connection-injected TEMPO_BASE_URL.

Output Rules

• TOKEN EFFICIENCY: Target <=50 lines per output
• Extract only needed span fields with jq
• Summarize trace data — never dump full trace payloads

Core Helper Function

#!/bin/bash

tempo_api() {
    local endpoint="$1"
    curl -s \
        -H "X-Scope-OrgID: ${TEMPO_TENANT_ID:-default}" \
        "${TEMPO_BASE_URL}${endpoint}"
}

tempo_search() {
    local traceql="$1"
    local limit="${2:-20}"
    local start="${3:-$(( $(date +%s) - 3600 ))}"
    local end="${4:-$(date +%s)}"

    tempo_api "/api/search?q=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${traceql}'))")&limit=${limit}&start=${start}&end=${end}"
}

tempo_trace() {
    local trace_id="$1"
    tempo_api "/api/traces/${trace_id}"
}

Parallel Execution

{
    tempo_api "/api/search/tags" &
    tempo_api "/status/buildinfo" &
    tempo_api "/api/echo" &
}
wait

Anti-Hallucination Rules

NEVER assume span attributes, service names, or resource attributes. ALWAYS discover first.

Phase 1: Discovery

#!/bin/bash
echo "=== Available Tags ==="
tempo_api "/api/search/tags" | jq -r '.tagNames[]' | sort | head -30

echo ""
echo "=== Service Names ==="
tempo_api "/api/search/tag/service.name/values" | jq -r '.tagValues[]' | sort

echo ""
echo "=== Available Span Attributes ==="
for tag in "http.method" "http.status_code" "span.kind" "status.code"; do
    values=$(tempo_api "/api/search/tag/${tag}/values" | jq -r '.tagValues[]' 2>/dev/null | head -5 | tr '\n' ',')
    [ -n "$values" ] && echo "$tag: $values"
done

Common Operations

TraceQL Search

#!/bin/bash
echo "=== Error Traces (last 1h) ==="
tempo_search '{ status = error }' 20 \
    | jq -r '.traces[] | "\(.traceID)\t\(.rootServiceName)\t\(.rootTraceName)\t\(.durationMs)ms"' \
    | head -20

echo ""
echo "=== Slow Traces (>1s) ==="
tempo_search '{ duration > 1s }' 20 \
    | jq -r '.traces[] | "\(.traceID)\t\(.rootServiceName)\t\(.rootTraceName)\t\(.durationMs)ms"' \
    | sort -t$'\t' -k4 -rn | head -15

echo ""
echo "=== Traces by Service ==="
SERVICE="${1:?Service name required}"
tempo_search "{ resource.service.name = \"${SERVICE}\" }" 20 \
    | jq -r '.traces[] | "\(.traceID)\t\(.durationMs)ms\t\(.rootTraceName)"' | head -15

Service Graph Analysis

#!/bin/bash
echo "=== Service Graph Metrics ==="
tempo_api "/api/metrics/service-graph" 2>/dev/null \
    | jq -r '.[] | "\(.client) -> \(.server)\treq_rate:\(.rate)\terr_rate:\(.errRate)\tp99:\(.p99)ms"' \
    | head -20

echo ""
echo "=== Service List ==="
tempo_api "/api/search/tag/service.name/values" \
    | jq -r '.tagValues[]' | while read svc; do
    count=$(tempo_search "{ resource.service.name = \"${svc}\" }" 1 | jq '.traces | length')
    echo "$svc: recent traces found=$count"
done

Span Metrics & Latency Analysis

#!/bin/bash
SERVICE="${1:?Service name required}"

echo "=== Span Metrics for ${SERVICE} ==="
tempo_search "{ resource.service.name = \"${SERVICE}\" }" 50 \
    | jq -r '{
        total_traces: (.traces | length),
        avg_duration_ms: ([.traces[].durationMs] | add / length | . * 10 | round / 10),
        max_duration_ms: ([.traces[].durationMs] | max),
        min_duration_ms: ([.traces[].durationMs] | min),
        error_traces: ([.traces[] | select(.rootServiceName != null)] | length)
    }'

echo ""
echo "=== Latency Distribution ==="
tempo_search "{ resource.service.name = \"${SERVICE}\" }" 100 \
    | jq -r '.traces | sort_by(.durationMs) | [
        "p50: \(.[length/2 | floor].durationMs)ms",
        "p90: \(.[length*9/10 | floor].durationMs)ms",
        "p99: \(.[length*99/100 | floor].durationMs)ms"
    ][]'

Trace Detail Inspection

#!/bin/bash
TRACE_ID="${1:?Trace ID required}"

echo "=== Trace Breakdown ==="
tempo_trace "$TRACE_ID" \
    | jq -r '.batches[].scopeSpans[].spans[] | "\(.name)\t\((.endTimeUnixNano - .startTimeUnixNano) / 1000000)ms\t\(.status.code // "OK")"' \
    | head -25

echo ""
echo "=== Error Spans ==="
tempo_trace "$TRACE_ID" \
    | jq -r '.batches[].scopeSpans[].spans[] | select(.status.code == 2) | "\(.name)\t\(.status.message // "no message")"'

Backend Health

#!/bin/bash
echo "=== Tempo Status ==="
{
    tempo_api "/status/buildinfo" | jq '{version, goVersion}' &
    tempo_api "/api/echo" > /dev/null && echo "Echo: OK" || echo "Echo: FAILED" &
    echo "=== Ready Check ==="
    curl -s "${TEMPO_BASE_URL}/ready" &
}
wait

Output Format

Present results as a structured report:

Monitoring Tempo Report
═══════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Counter-Rationalizations

Common Pitfalls

• TraceQL syntax: Uses { } braces for span selectors — { status = error } not status = error
• Duration format: TraceQL uses human-readable — { duration > 500ms }, { duration > 2s }
• Attribute types: String values need quotes — { resource.service.name = "api" } not = api
• Tag discovery: Always use /api/search/tags and /api/search/tag/{tag}/values before filtering
• OTLP format: Trace responses use OTLP proto format — timestamps are nanoseconds
• Status codes: 0=Unset, 1=OK, 2=Error in OTLP — filter errors with .status.code == 2
• Multi-tenancy: Set X-Scope-OrgID header for multi-tenant deployments
• Search limits: Default search window is limited — always specify start and end timestamps