Unified Observability: OpenTelemetry, Prometheus & Distributed Tracing
When to Use This Skill
Triggers — load this skill when:
- A service or platform needs metrics, logs, and traces wired end to end
- Trace sampling, exemplars, or log-to-trace correlation must be configured
- Cardinality or collector resource problems are degrading the telemetry stack
Route elsewhere when:
- Objective and burn-rate alert definition ->
sli-slo-error-budget-design
- Node/container infrastructure metrics ->
infrastructure-host-monitoring
1. OpenTelemetry Collector Pipeline Configuration (otel-collector.yaml)
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 20
resourcedetection:
detectors: [env, gcp, ecs, ec2, azure]
timeout: 2s
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
namespace: "otel"
otlp/tempo:
endpoint: "tempo.monitoring:4317"
tls:
insecure: true
loki:
endpoint: "http://loki.monitoring:3100/loki/api/v1/push"
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch, resourcedetection]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [loki]
2. Distributed Tracing Best Practices
- Trace Context Propagation: Pass
traceparent headers via W3C TraceContext standards across all inter-service REST, gRPC, and Kafka messages.
- Span Attributes: Enrich spans with high-cardinality metadata (
customer_id, order_id, http.status_code).
- Tail-based Sampling: Sample 100% of error traces and slow traces (> 95th percentile latency) while dropping 90% of routine fast transactions to control storage costs.
3. Anti-Patterns
| Anti-pattern |
Why it fails in production |
| Putting a user ID, request ID, email or full URL in a metric label |
Cardinality is multiplicative: one unbounded label multiplied by an existing one takes Prometheus OOM in hours. High-cardinality identifiers belong on traces and logs, never on metrics. |
| Sampling traces at the SDK with a fixed low rate |
The 1% you keep is random, so the slow and failing requests you actually needed are gone. Use tail sampling in the Collector, keeping all errors and slow traces plus a small baseline of successes. |
| Instrumenting every service with a different SDK config |
Inconsistent resource attributes (service.name, deployment.environment) make cross-service queries impossible. Standardise resource attributes centrally and inject them via the Collector. |
| Logging without a trace ID |
Three services' logs stay three unrelated streams, and every incident starts with manual correlation by timestamp. Propagate traceparent and emit trace_id on every structured log line. |
| Running the Collector as a single deployment for everything |
One noisy service's spans starve the pipeline for all others. Split agent (per-node, cheap) from gateway (central, batching and tail sampling), and set memory limiter processors. |
| Dashboards built per team with no shared definitions |
"Latency" means p50 on one dashboard and mean on another; incident calls argue about the data instead of the outage. Define recording rules once and build dashboards from them. |
1---2name: prometheus-grafana-otel-tracing3description: Unified observability: OpenTelemetry Collector pipelines, Prometheus scraping and recording rules, Loki log correlation, Tempo distributed tracing, exemplars, sampling strategy, and cardinality control. Use when instrumenting services so a latency spike can be followed to the exact trace and log line, building the metrics-logs-traces stack, or fixing missing telemetry and cardinality blowups.4---56# Unified Observability: OpenTelemetry, Prometheus & Distributed Tracing78## When to Use This Skill910**Triggers — load this skill when:**1112- A service or platform needs metrics, logs, and traces wired end to end13- Trace sampling, exemplars, or log-to-trace correlation must be configured14- Cardinality or collector resource problems are degrading the telemetry stack1516**Route elsewhere when:**1718- Objective and burn-rate alert definition -> `sli-slo-error-budget-design`19- Node/container infrastructure metrics -> `infrastructure-host-monitoring`2021## 1. OpenTelemetry Collector Pipeline Configuration (`otel-collector.yaml`)2223```yaml24receivers:25 otlp:26 protocols:27 grpc:28 endpoint: 0.0.0.0:431729 http:30 endpoint: 0.0.0.0:43183132processors:33 batch:34 timeout: 1s35 send_batch_size: 102436 memory_limiter:37 check_interval: 1s38 limit_percentage: 7539 spike_limit_percentage: 2040 resourcedetection:41 detectors: [env, gcp, ecs, ec2, azure]42 timeout: 2s4344exporters:45 prometheus:46 endpoint: "0.0.0.0:8889"47 namespace: "otel"48 otlp/tempo:49 endpoint: "tempo.monitoring:4317"50 tls:51 insecure: true52 loki:53 endpoint: "http://loki.monitoring:3100/loki/api/v1/push"5455service:56 pipelines:57 traces:58 receivers: [otlp]59 processors: [memory_limiter, batch, resourcedetection]60 exporters: [otlp/tempo]61 metrics:62 receivers: [otlp]63 processors: [memory_limiter, batch]64 exporters: [prometheus]65 logs:66 receivers: [otlp]67 processors: [memory_limiter, batch]68 exporters: [loki]69```7071---7273## 2. Distributed Tracing Best Practices7475- **Trace Context Propagation**: Pass `traceparent` headers via W3C TraceContext standards across all inter-service REST, gRPC, and Kafka messages.76- **Span Attributes**: Enrich spans with high-cardinality metadata (`customer_id`, `order_id`, `http.status_code`).77- **Tail-based Sampling**: Sample 100% of error traces and slow traces (> 95th percentile latency) while dropping 90% of routine fast transactions to control storage costs.7879---8081## 3. Anti-Patterns8283| Anti-pattern | Why it fails in production |84| --- | --- |85| Putting a user ID, request ID, email or full URL in a metric label | Cardinality is multiplicative: one unbounded label multiplied by an existing one takes Prometheus OOM in hours. High-cardinality identifiers belong on traces and logs, never on metrics. |86| Sampling traces at the SDK with a fixed low rate | The 1% you keep is random, so the slow and failing requests you actually needed are gone. Use tail sampling in the Collector, keeping all errors and slow traces plus a small baseline of successes. |87| Instrumenting every service with a different SDK config | Inconsistent resource attributes (`service.name`, `deployment.environment`) make cross-service queries impossible. Standardise resource attributes centrally and inject them via the Collector. |88| Logging without a trace ID | Three services' logs stay three unrelated streams, and every incident starts with manual correlation by timestamp. Propagate `traceparent` and emit `trace_id` on every structured log line. |89| Running the Collector as a single deployment for everything | One noisy service's spans starve the pipeline for all others. Split agent (per-node, cheap) from gateway (central, batching and tail sampling), and set memory limiter processors. |90| Dashboards built per team with no shared definitions | "Latency" means p50 on one dashboard and mean on another; incident calls argue about the data instead of the outage. Define recording rules once and build dashboards from them. |
mchittineni