# Observability Stack > Observability architecture with OpenTelemetry, Prometheus, Grafana, and structured logging. Covers RED/USE metrics, SLI/SLO monitoring, multi-window burn-rate alerting, dashboard design, and distributed tracing. Auto-activates on monitoring, observability, metrics, alerting, dashboard, Prometheus, Grafana, OpenTelemetry, tracing, logging, SLI, SLO. - Skill: `pfangueiro/observability-stack` (Agent Skill) - Install (CLI): `npx skillmds@latest add pfangueiro/observability-stack` - Raw SKILL.md: https://api.skillmd.com/api/skills/pfangueiro/observability-stack/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: pfangueiro (https://skillmd.com/u/pfangueiro) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/pfangueiro/observability-stack --- # Observability Stack ## Overview Production observability using the three pillars (metrics, logs, traces) with OpenTelemetry as the collection standard. Covers metric design, alerting strategies, dashboard patterns, and cost management. ## The Three Pillars | Pillar | What | Tool Stack | When | |--------|------|-----------|------| | **Metrics** | Numeric time-series data | Prometheus + Grafana | Trend detection, alerting, capacity | | **Logs** | Structured event records | Loki / ELK / CloudWatch | Debugging, audit trails, forensics | | **Traces** | Request flow across services | Tempo / Jaeger / X-Ray | Latency analysis, dependency mapping | All three must be **correlated** via trace context (trace_id, span_id) for effective debugging. ## Metric Design ### RED Method (Service-Level) Measure what **users experience**. Apply to every service endpoint. ```promql # Rate — requests per second rate(http_requests_total{service="api"}[5m]) # Errors — error percentage rate(http_requests_total{service="api", status=~"5.."}[5m]) / rate(http_requests_total{service="api"}[5m]) * 100 # Duration — p99 latency histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="api"}[5m])) ``` ### USE Method (Infrastructure-Level) Measure what the **infrastructure is doing**. Apply to every resource. | Resource | Utilization | Saturation | Errors | |----------|------------|------------|--------| | **CPU** | `node_cpu_seconds_total` | `node_load15` / num_cpus | — | | **Memory** | `node_memory_MemAvailable_bytes` | `node_memory_SwapFree_bytes` | OOM events | | **Disk** | `node_filesystem_avail_bytes` | `node_disk_io_time_weighted_seconds_total` | `node_disk_io_errors` | | **Network** | `node_network_transmit_bytes_total` | `node_network_transmit_drop_total` | `node_network_transmit_errs_total` | | **K8s Pod** | CPU/mem requests vs limits | Throttling, evictions | CrashLoopBackOff count | ### Naming Convention Follow Prometheus naming: `___` ``` http_requests_total # counter — always _total suffix http_request_duration_seconds # histogram — always include unit http_requests_in_flight # gauge — current value ``` ## SLI/SLO Monitoring ### Define SLIs as Prometheus Recording Rules ```yaml groups: - name: sli_rules # Evaluate no faster than 1m: the 3d-range rule below is expensive to recompute. interval: 1m rules: # Every SLI is grouped `by (service)` — the dimension the SLO is defined on. # # An ungrouped `sum(rate(...))` collapses the whole fleet into one number and # DILUTES a single service's outage below every burn-rate threshold. With nine # healthy services at 100 req/min and one 10 req/min service that is 100% down, # the fleet error ratio is 10/910 = 0.011 — under the 0.0144 fast-burn # threshold. The service is completely dead and nobody is paged. # # The `or 0 * ...` fallback is not decoration. Binary operators drop series with # no match on the other side, so a service serving *only* 5xx has no # `status!~"5.."` numerator, is dropped by the division, and vanishes from the # SLI — the total-outage case is precisely the one that disappears. The fallback # pins those services to 0 instead of absent. Grouping alone does NOT fix this; # both halves are required. (Verified with `promtool test rules` — see below.) # Availability SLI: ratio of successful requests, per service - record: sli:availability:ratio_rate5m expr: | ( sum by (service) (rate(http_requests_total{status!~"5.."}[5m])) or 0 * sum by (service) (rate(http_requests_total[5m])) ) / sum by (service) (rate(http_requests_total[5m])) # Long-window availability SLIs — REQUIRED by the multi-window burn-rate # alerts below. Every window an alert references must be recorded here: # Prometheus returns an empty vector for an unknown metric name, so an # alert ANDing an undefined rule silently never fires and logs no error. # Keep the grouping identical across windows: `and` matches on the full label # set, so a rule grouped differently from its partner never intersects. - record: sli:availability:ratio_rate30m expr: | ( sum by (service) (rate(http_requests_total{status!~"5.."}[30m])) or 0 * sum by (service) (rate(http_requests_total[30m])) ) / sum by (service) (rate(http_requests_total[30m])) - record: sli:availability:ratio_rate1h expr: | ( sum by (service) (rate(http_requests_total{status!~"5.."}[1h])) or 0 * sum by (service) (rate(http_requests_total[1h])) ) / sum by (service) (rate(http_requests_total[1h])) - record: sli:availability:ratio_rate6h expr: | ( sum by (service) (rate(http_requests_total{status!~"5.."}[6h])) or 0 * sum by (service) (rate(http_requests_total[6h])) ) / sum by (service) (rate(http_requests_total[6h])) # 3d window needs >= 3d retention; evaluate this group no faster than # `interval: 1m` — long-range rate() is expensive to recompute. - record: sli:availability:ratio_rate3d expr: | ( sum by (service) (rate(http_requests_total{status!~"5.."}[3d])) or 0 * sum by (service) (rate(http_requests_total[3d])) ) / sum by (service) (rate(http_requests_total[3d])) # Latency SLI: ratio of requests faster than 300ms, per service - record: sli:latency:ratio_rate5m expr: | ( sum by (service) (rate(http_request_duration_seconds_bucket{le="0.3"}[5m])) or 0 * sum by (service) (rate(http_request_duration_seconds_count[5m])) ) / sum by (service) (rate(http_request_duration_seconds_count[5m])) ``` ### Error Budget Calculation ```promql # Error budget remaining (30-day window), for one service. # The SLI is recorded `by (service)`, so name the service you mean — left # unqualified this returns one series per service, never a fleet-wide number. 1 - ( (1 - avg_over_time(sli:availability:ratio_rate5m{service="api"}[30d])) / (1 - 0.999) # SLO target ) ``` ### Multi-Window Multi-Burn-Rate Alerts Google SRE recommended approach — alert on budget consumption rate, not raw thresholds. ```yaml groups: - name: slo_alerts # Each operand below is a per-service vector (the SLIs are recorded # `by (service)`), and `1 - ` keeps the vector's labels while dropping # __name__. Both sides of every `and` therefore carry exactly `{service="..."}` # and intersect per service, so ONE service breaching its own budget pages on # its own. If the two windows in an `and` are ever grouped differently, the # label sets stop matching and the alert can never fire. rules: # Fast burn: 2% budget consumed in 1 hour → page - alert: SLOBurnRateCritical expr: | ( 1 - sli:availability:ratio_rate1h > (14.4 * (1 - 0.999)) and 1 - sli:availability:ratio_rate5m > (14.4 * (1 - 0.999)) ) for: 2m labels: severity: critical annotations: summary: "High error budget burn rate (fast burn) for service {{ $labels.service }}" runbook_url: "https://wiki/runbooks/slo-burn-rate" # Slow burn: 5% budget consumed in 6 hours → page - alert: SLOBurnRateWarning expr: | ( 1 - sli:availability:ratio_rate6h > (6 * (1 - 0.999)) and 1 - sli:availability:ratio_rate30m > (6 * (1 - 0.999)) ) for: 15m labels: severity: warning annotations: summary: "Elevated error budget burn rate (slow burn) for service {{ $labels.service }}" runbook_url: "https://wiki/runbooks/slo-burn-rate" # Gradual burn: 10% budget consumed in 3 days → ticket - alert: SLOBurnRateTicket expr: | ( 1 - sli:availability:ratio_rate3d > (1 * (1 - 0.999)) and 1 - sli:availability:ratio_rate6h > (1 * (1 - 0.999)) ) for: 1h labels: severity: ticket # A ticket-severity alert reaches someone with no pager context and no # incident channel, so it needs MORE explanation than the page, not less. # Annotations are what turns a row in a queue into something actionable. annotations: summary: "Gradual error budget burn for service {{ $labels.service }} — 10% of the 30d budget in 3 days" description: >- 3d error ratio for {{ $labels.service }} is {{ $value | humanizePercentage }}, above the 1x sustainable burn for a 99.9% SLO. Not urgent, but at this pace the 30d budget is exhausted before the window closes. Triage in business hours. runbook_url: "https://wiki/runbooks/slo-burn-rate" # Companion to all three burn-rate alerts: catches the outage a RATIO cannot # see (see "Zero Traffic Is the Blind Spot" below). Self-inventorying — the # long left-hand window is the list of services that are supposed to be serving. # left = served traffic at some point in the last day # right = serving traffic now # `unless` keeps left-hand elements with no exactly-matching label set on the # right, so a service that went flat (rate 0, filtered out by `> 0`) OR vanished # (no series at all in the 10m window) survives the filter and fires. - alert: ServiceTrafficAbsent expr: | ( sum by (service) (rate(http_requests_total[1d])) > 0 unless sum by (service) (rate(http_requests_total[10m])) > 0 ) for: 10m labels: severity: critical annotations: summary: "Service {{ $labels.service }} is serving no traffic — its SLO alerts are blind, not green" description: >- No requests in 10m for a service that served traffic within the last day. The burn-rate alerts above divide by zero here and cannot fire, so treat a quiet SLO dashboard for {{ $labels.service }} as unverified, not healthy. runbook_url: "https://wiki/runbooks/traffic-absent" ``` ### Zero Traffic Is the Blind Spot in Every Ratio SLI A burn-rate alert divides errors by requests, so it has nothing to say when there are no requests. Two shapes of **total** outage therefore read as perfectly healthy: | Failure | What the SLI evaluates to | Why the alert stays silent | |---|---|---| | Process up, receiving nothing (load balancer misroute, upstream dead, consumer stopped) | numerator and denominator are both `rate() == 0`, so the ratio is `0 / 0` | Prometheus arithmetic follows [IEEE 754](https://prometheus.io/docs/prometheus/latest/querying/operators/), so `0/0` is `NaN` and `1 - NaN` is `NaN`. Every ordered comparison against `NaN` is false, so `> threshold` never matches and the alert has no series to fire on | | Target gone entirely (pod deleted, job dropped from discovery) | selectors match nothing, so the recording rule emits **no series at all** | An alert over an empty vector never fires and logs nothing; the panel renders "No data" | The `or 0 * sum(...)` guard in the recording rules above does **not** cover this. It supplies a `0` numerator when a service served no *non-5xx* requests. It cannot supply a denominator when the service served no requests at all. That is why `ServiceTrafficAbsent` is in the group above rather than described here as an optional extra — copying the burn-rate rules without it ships the blind spot. Know its edges: - **The self-inventory expires.** The `[1d]` left-hand window is the alert's entire memory of which services exist. Once a service has been silent longer than that window it drops off the left-hand side, the alert **resolves itself**, and the outage carries on unwatched. Set that window to the longest outage you must still be paged for, and pay the retention cost. - **A service that has never served traffic is invisible** — there is no left-hand element to keep. For a service you must never lose, name it explicitly: ```promql absent(http_requests_total{service="checkout"}) ``` `absent()` returns a 1-element vector only when its argument matches nothing, and it derives the output labels from **equality matchers only** — `absent(sum(...))` returns a label-less `{}`, and a `=~` matcher is dropped. Keep it a plain selector, and add one rule per service you refuse to lose silently. - **`up == 0` is not a substitute.** That reports a target Prometheus cannot scrape; this reports a healthy, scraped process that nothing is talking to. They fail independently — alert on both. - **Low traffic breaks the ratio too, in the opposite direction.** At 5 requests in a window a single error is a 20% error rate, and fast burn trips on statistical noise. Either add a minimum-volume term to the burn-rate `and` chain — `and sum by (service) (rate(http_requests_total[5m])) > 0.1` (≈30 requests per 5m) — or move low-volume services off ratio alerting onto an absolute error-count alert. Adding the volume term makes the alert *more* dependent on `ServiceTrafficAbsent`, since it explicitly excludes the quiet case. Like every other alert here, this one is only real once a unit test proves it fires (next section). Drive it with a series that stops mid-test — `values: '0+100x30 _x60'`, where [`_` is a missing sample](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/) — and assert `ServiceTrafficAbsent` in `exp_alerts` after the gap. ### Verify the Rules Actually Fire **A silent alert and a healthy service look identical.** A PromQL selector naming a metric that was never recorded — a typo, a deleted recording rule, a window nobody recorded — returns an **empty vector**. Empty is not an error: the alert evaluates to zero series, never fires, logs nothing, and the panel renders "No data", which reads as green. The failure mode of a broken SLO alert is silence, and silence is exactly what a healthy system produces. `promtool check rules` **cannot catch this.** It parses and type-checks rule files with no TSDB attached, so it has no way to know whether any series ever satisfies a selector. It passes an ungrouped, fleet-diluted SLI and a misspelled metric name with the same cheerful `SUCCESS`. Treat exit 0 as "the YAML is well-formed", never as "this alert can fire". The only check that catches both is a [rule unit test](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/), which synthesizes a TSDB and asserts on the alerts produced: ```yaml # slo_test.yml rule_files: - sli_rules.yml - slo_alerts.yml evaluation_interval: 1m tests: - interval: 1m input_series: # One busy healthy service... - series: 'http_requests_total{service="api", status="200"}' values: '0+1000x120' # ...and one small service that is 100% down. Fleet-wide the error ratio is # only 10/1010 = 0.0099, UNDER the 0.0144 fast-burn threshold — so with an # ungrouped SLI this test fails with `got:[]` and checkout dies unnoticed. - series: 'http_requests_total{service="checkout", status="500"}' values: '0+10x120' alert_rule_test: - eval_time: 90m alertname: SLOBurnRateCritical exp_alerts: - exp_labels: severity: critical service: checkout exp_annotations: summary: "High error budget burn rate (fast burn) for service checkout" runbook_url: "https://wiki/runbooks/slo-burn-rate" ``` ```bash promtool check rules sli_rules.yml slo_alerts.yml # syntax only — proves nothing about firing promtool test rules slo_test.yml # actually evaluates the alert ``` Run both in CI. Every SLO alert needs at least one unit test that would fail if the alert stopped firing — that is the only way the "Tested" box below is honest. ## Alert Quality Checklist Every alert MUST have: - [ ] **Runbook link** in annotations - [ ] **Clear summary** describing what is wrong and impact - [ ] **Severity** matching response expectation (page vs ticket vs log) - [ ] **Owner** — which team is responsible - [ ] **Actionable** — recipient can do something about it - [ ] **Tested** — alert has fired in non-prod and was validated Alert anti-patterns to avoid: - Alerting on raw CPU% instead of SLO burn rate - Flapping alerts (fire and resolve repeatedly) - Alerts with no runbook ("figure it out yourself") - Duplicate alerts for the same condition from different sources - Alerts nobody looks at (alert fatigue → ignored pages) ## Dashboard Design ### Layer 1: Service Overview (start here) One dashboard per team showing all owned services: - Traffic (req/sec) per service - Error rate per service (color: green <0.1%, yellow <1%, red >1%) - p99 latency per service - Error budget remaining per service (30-day window) ### Layer 2: Per-Service Deep Dive - RED metrics broken down by endpoint and status code - Dependency health (upstream/downstream latencies) - Recent deployments overlay on graphs - Top errors table with counts and sample traces ### Layer 3: Infrastructure - USE metrics per node/pod - Kubernetes resource utilization vs requests vs limits - Pod restart counts, eviction counts - Node health, disk pressure, memory pressure ### Layer 4: Business Metrics - Conversion rate, cart abandonment, signup completion - Revenue per minute (correlated with system health) - Feature flag adoption rates ## OpenTelemetry Instrumentation ### Auto-Instrumentation (Start Here) ```bash # Node.js — auto-instrument without code changes npm install @opentelemetry/auto-instrumentations-node node --require @opentelemetry/auto-instrumentations-node/register app.js # Python — auto-instrument without code changes pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-bootstrap -a install opentelemetry-instrument python app.py ``` ### Manual Spans (Add Business Context) ```typescript import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer('payment-service'); async function processPayment(order: Order) { return tracer.startActiveSpan('process-payment', async (span) => { span.setAttribute('order.id', order.id); span.setAttribute('order.amount', order.amount); span.setAttribute('payment.method', order.paymentMethod); try { const result = await chargeCard(order); span.setAttribute('payment.status', 'success'); return result; } catch (error) { span.recordException(error); span.setStatus({ code: SpanStatusCode.ERROR }); throw error; } finally { span.end(); } }); } ``` ### Collector Configuration ```yaml # otel-collector-config.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: timeout: 5s send_batch_size: 1000 memory_limiter: check_interval: 1s limit_mib: 512 exporters: prometheus: endpoint: 0.0.0.0:8889 # Loki ingests OTLP natively, so push logs with the core `otlphttp` exporter. # It appends the signal path itself: this posts to http://loki:3100/otlp/v1/logs. # # Do NOT use the old `loki` exporter here. It was deprecated on 2024-07-09 and # DELETED from opentelemetry-collector-contrib in v0.131.0 (2025-07-29), and # collector config resolution is ALL-OR-NOTHING: one unknown exporter type fails # the whole unmarshal at startup — `'exporters' unknown type: "loki" for id: # "loki"` — and the process never comes up, so the failure happens before any # pipeline is built. A stale logs exporter therefore takes the metrics and # traces pipelines down with it — you lose all three signals, not just logs. otlphttp/loki: endpoint: http://loki:3100/otlp otlp/tempo: endpoint: tempo:4317 tls: insecure: true service: pipelines: metrics: receivers: [otlp] processors: [memory_limiter, batch] exporters: [prometheus] logs: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlphttp/loki] traces: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp/tempo] ``` ## Structured Logging ```json { "timestamp": "2026-03-21T10:30:00.000Z", "level": "error", "service": "payment-api", "trace_id": "abc123def456", "span_id": "789ghi", "message": "Payment processing failed", "error.type": "CardDeclinedException", "error.message": "Insufficient funds", "order.id": "ord-12345", "http.method": "POST", "http.path": "/api/payments", "http.status_code": 402, "duration_ms": 234 } ``` Rules: - Always JSON, never unstructured text in production - Include trace_id and span_id for correlation - Use consistent field names (follow OTel semantic conventions) - Redact PII: credit card numbers, passwords, tokens, email addresses - Log levels: ERROR (action needed), WARN (degraded), INFO (significant events), DEBUG (dev only, never in prod) ## Cost Management Observability can become expensive. Control costs with: - **Sampling**: Use head-based (10% of traces) or tail-based (100% of errors/slow) in OTel Collector - **Retention**: 15 days for metrics, 7 days for logs, 3 days for traces (adjust per need) - **Cardinality**: Never use user_id, request_id, or session_id as metric labels - **Aggregation**: Pre-aggregate in recording rules, query aggregated data - **Tiering**: Hot (recent, fast queries) → Warm (older, slower) → Cold (archive, S3)