Observability Stack

Observability Stack

Overview

Production observability using the three pillars (metrics, logs, traces) with OpenTelemetry as the collection standard. Covers metric design, alerting strategies, dashboard patterns, and cost management.

The Three Pillars

All three must be correlated via trace context (trace_id, span_id) for effective debugging.

Metric Design

RED Method (Service-Level)

Measure what users experience. Apply to every service endpoint.

# Rate — requests per second
rate(http_requests_total{service="api"}[5m])

# Errors — error percentage
rate(http_requests_total{service="api", status=~"5.."}[5m])
/ rate(http_requests_total{service="api"}[5m]) * 100

# Duration — p99 latency
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="api"}[5m]))

USE Method (Infrastructure-Level)

Measure what the infrastructure is doing. Apply to every resource.

Naming Convention

Follow Prometheus naming: <namespace>_<name>_<unit>_<suffix>

http_requests_total           # counter — always _total suffix
http_request_duration_seconds # histogram — always include unit
http_requests_in_flight       # gauge — current value

SLI/SLO Monitoring

Define SLIs as Prometheus Recording Rules

groups:
  - name: sli_rules
    # Evaluate no faster than 1m: the 3d-range rule below is expensive to recompute.
    interval: 1m
    rules:
      # Every SLI is grouped `by (service)` — the dimension the SLO is defined on.
      #
      # An ungrouped `sum(rate(...))` collapses the whole fleet into one number and
      # DILUTES a single service's outage below every burn-rate threshold. With nine
      # healthy services at 100 req/min and one 10 req/min service that is 100% down,
      # the fleet error ratio is 10/910 = 0.011 — under the 0.0144 fast-burn
      # threshold. The service is completely dead and nobody is paged.
      #
      # The `or 0 * ...` fallback is not decoration. Binary operators drop series with
      # no match on the other side, so a service serving *only* 5xx has no
      # `status!~"5.."` numerator, is dropped by the division, and vanishes from the
      # SLI — the total-outage case is precisely the one that disappears. The fallback
      # pins those services to 0 instead of absent. Grouping alone does NOT fix this;
      # both halves are required. (Verified with `promtool test rules` — see below.)

      # Availability SLI: ratio of successful requests, per service
      - record: sli:availability:ratio_rate5m
        expr: |
          (
            sum by (service) (rate(http_requests_total{status!~"5.."}[5m]))
            or 0 * sum by (service) (rate(http_requests_total[5m]))
          )
          / sum by (service) (rate(http_requests_total[5m]))

      # Long-window availability SLIs — REQUIRED by the multi-window burn-rate
      # alerts below. Every window an alert references must be recorded here:
      # Prometheus returns an empty vector for an unknown metric name, so an
      # alert ANDing an undefined rule silently never fires and logs no error.
      # Keep the grouping identical across windows: `and` matches on the full label
      # set, so a rule grouped differently from its partner never intersects.
      - record: sli:availability:ratio_rate30m
        expr: |
          (
            sum by (service) (rate(http_requests_total{status!~"5.."}[30m]))
            or 0 * sum by (service) (rate(http_requests_total[30m]))
          )
          / sum by (service) (rate(http_requests_total[30m]))

      - record: sli:availability:ratio_rate1h
        expr: |
          (
            sum by (service) (rate(http_requests_total{status!~"5.."}[1h]))
            or 0 * sum by (service) (rate(http_requests_total[1h]))
          )
          / sum by (service) (rate(http_requests_total[1h]))

      - record: sli:availability:ratio_rate6h
        expr: |
          (
            sum by (service) (rate(http_requests_total{status!~"5.."}[6h]))
            or 0 * sum by (service) (rate(http_requests_total[6h]))
          )
          / sum by (service) (rate(http_requests_total[6h]))

      # 3d window needs >= 3d retention; evaluate this group no faster than
      # `interval: 1m` — long-range rate() is expensive to recompute.
      - record: sli:availability:ratio_rate3d
        expr: |
          (
            sum by (service) (rate(http_requests_total{status!~"5.."}[3d]))
            or 0 * sum by (service) (rate(http_requests_total[3d]))
          )
          / sum by (service) (rate(http_requests_total[3d]))

      # Latency SLI: ratio of requests faster than 300ms, per service
      - record: sli:latency:ratio_rate5m
        expr: |
          (
            sum by (service) (rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
            or 0 * sum by (service) (rate(http_request_duration_seconds_count[5m]))
          )
          / sum by (service) (rate(http_request_duration_seconds_count[5m]))

Error Budget Calculation

# Error budget remaining (30-day window), for one service.
# The SLI is recorded `by (service)`, so name the service you mean — left
# unqualified this returns one series per service, never a fleet-wide number.
1 - (
  (1 - avg_over_time(sli:availability:ratio_rate5m{service="api"}[30d]))
  / (1 - 0.999)  # SLO target
)

Multi-Window Multi-Burn-Rate Alerts

Google SRE recommended approach — alert on budget consumption rate, not raw thresholds.

groups:
  - name: slo_alerts
    # Each operand below is a per-service vector (the SLIs are recorded
    # `by (service)`), and `1 - <vector>` keeps the vector's labels while dropping
    # __name__. Both sides of every `and` therefore carry exactly `{service="..."}`
    # and intersect per service, so ONE service breaching its own budget pages on
    # its own. If the two windows in an `and` are ever grouped differently, the
    # label sets stop matching and the alert can never fire.
    rules:
      # Fast burn: 2% budget consumed in 1 hour → page
      - alert: SLOBurnRateCritical
        expr: |
          (
            1 - sli:availability:ratio_rate1h > (14.4 * (1 - 0.999))
            and
            1 - sli:availability:ratio_rate5m > (14.4 * (1 - 0.999))
          )
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error budget burn rate (fast burn) for service {{ $labels.service }}"
          runbook_url: "https://wiki/runbooks/slo-burn-rate"

      # Slow burn: 5% budget consumed in 6 hours → page
      - alert: SLOBurnRateWarning
        expr: |
          (
            1 - sli:availability:ratio_rate6h > (6 * (1 - 0.999))
            and
            1 - sli:availability:ratio_rate30m > (6 * (1 - 0.999))
          )
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Elevated error budget burn rate (slow burn) for service {{ $labels.service }}"
          runbook_url: "https://wiki/runbooks/slo-burn-rate"

      # Gradual burn: 10% budget consumed in 3 days → ticket
      - alert: SLOBurnRateTicket
        expr: |
          (
            1 - sli:availability:ratio_rate3d > (1 * (1 - 0.999))
            and
            1 - sli:availability:ratio_rate6h > (1 * (1 - 0.999))
          )
        for: 1h
        labels:
          severity: ticket
        # A ticket-severity alert reaches someone with no pager context and no
        # incident channel, so it needs MORE explanation than the page, not less.
        # Annotations are what turns a row in a queue into something actionable.
        annotations:
          summary: "Gradual error budget burn for service {{ $labels.service }} — 10% of the 30d budget in 3 days"
          description: >-
            3d error ratio for {{ $labels.service }} is {{ $value | humanizePercentage }},
            above the 1x sustainable burn for a 99.9% SLO. Not urgent, but at this pace the
            30d budget is exhausted before the window closes. Triage in business hours.
          runbook_url: "https://wiki/runbooks/slo-burn-rate"

      # Companion to all three burn-rate alerts: catches the outage a RATIO cannot
      # see (see "Zero Traffic Is the Blind Spot" below). Self-inventorying — the
      # long left-hand window is the list of services that are supposed to be serving.
      #   left  = served traffic at some point in the last day
      #   right = serving traffic now
      # `unless` keeps left-hand elements with no exactly-matching label set on the
      # right, so a service that went flat (rate 0, filtered out by `> 0`) OR vanished
      # (no series at all in the 10m window) survives the filter and fires.
      - alert: ServiceTrafficAbsent
        expr: |
          (
            sum by (service) (rate(http_requests_total[1d])) > 0
            unless
            sum by (service) (rate(http_requests_total[10m])) > 0
          )
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.service }} is serving no traffic — its SLO alerts are blind, not green"
          description: >-
            No requests in 10m for a service that served traffic within the last day.
            The burn-rate alerts above divide by zero here and cannot fire, so treat a
            quiet SLO dashboard for {{ $labels.service }} as unverified, not healthy.
          runbook_url: "https://wiki/runbooks/traffic-absent"

Zero Traffic Is the Blind Spot in Every Ratio SLI

A burn-rate alert divides errors by requests, so it has nothing to say when there are no requests. Two shapes of total outage therefore read as perfectly healthy:

The or 0 * sum(...) guard in the recording rules above does not cover this. It supplies a 0 numerator when a service served no non-5xx requests. It cannot supply a denominator when the service served no requests at all.

That is why ServiceTrafficAbsent is in the group above rather than described here as an optional extra — copying the burn-rate rules without it ships the blind spot. Know its edges:

• The self-inventory expires. The [1d] left-hand window is the alert's entire memory of which services exist. Once a service has been silent longer than that window it drops off the left-hand side, the alert resolves itself, and the outage carries on unwatched. Set that window to the longest outage you must still be paged for, and pay the retention cost.

• A service that has never served traffic is invisible — there is no left-hand element to keep. For a service you must never lose, name it explicitly:

  absent(http_requests_total{service="checkout"})
  

  absent() returns a 1-element vector only when its argument matches nothing, and it derives the output labels from equality matchers only — absent(sum(...)) returns a label-less {}, and a =~ matcher is dropped. Keep it a plain selector, and add one rule per service you refuse to lose silently.

• up == 0 is not a substitute. That reports a target Prometheus cannot scrape; this reports a healthy, scraped process that nothing is talking to. They fail independently — alert on both.

• Low traffic breaks the ratio too, in the opposite direction. At 5 requests in a window a single error is a 20% error rate, and fast burn trips on statistical noise. Either add a minimum-volume term to the burn-rate and chain — and sum by (service) (rate(http_requests_total[5m])) > 0.1 (≈30 requests per 5m) — or move low-volume services off ratio alerting onto an absolute error-count alert. Adding the volume term makes the alert more dependent on ServiceTrafficAbsent, since it explicitly excludes the quiet case.

Like every other alert here, this one is only real once a unit test proves it fires (next section). Drive it with a series that stops mid-test — values: '0+100x30 _x60', where _ is a missing sample — and assert ServiceTrafficAbsent in exp_alerts after the gap.

Verify the Rules Actually Fire

A silent alert and a healthy service look identical. A PromQL selector naming a metric that was never recorded — a typo, a deleted recording rule, a window nobody recorded — returns an empty vector. Empty is not an error: the alert evaluates to zero series, never fires, logs nothing, and the panel renders "No data", which reads as green. The failure mode of a broken SLO alert is silence, and silence is exactly what a healthy system produces.

promtool check rules cannot catch this. It parses and type-checks rule files with no TSDB attached, so it has no way to know whether any series ever satisfies a selector. It passes an ungrouped, fleet-diluted SLI and a misspelled metric name with the same cheerful SUCCESS. Treat exit 0 as "the YAML is well-formed", never as "this alert can fire".

The only check that catches both is a rule unit test, which synthesizes a TSDB and asserts on the alerts produced:

# slo_test.yml
rule_files:
  - sli_rules.yml
  - slo_alerts.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      # One busy healthy service...
      - series: 'http_requests_total{service="api", status="200"}'
        values: '0+1000x120'
      # ...and one small service that is 100% down. Fleet-wide the error ratio is
      # only 10/1010 = 0.0099, UNDER the 0.0144 fast-burn threshold — so with an
      # ungrouped SLI this test fails with `got:[]` and checkout dies unnoticed.
      - series: 'http_requests_total{service="checkout", status="500"}'
        values: '0+10x120'
    alert_rule_test:
      - eval_time: 90m
        alertname: SLOBurnRateCritical
        exp_alerts:
          - exp_labels:
              severity: critical
              service: checkout
            exp_annotations:
              summary: "High error budget burn rate (fast burn) for service checkout"
              runbook_url: "https://wiki/runbooks/slo-burn-rate"

promtool check rules sli_rules.yml slo_alerts.yml  # syntax only — proves nothing about firing
promtool test rules slo_test.yml                   # actually evaluates the alert

Run both in CI. Every SLO alert needs at least one unit test that would fail if the alert stopped firing — that is the only way the "Tested" box below is honest.

Alert Quality Checklist

Every alert MUST have:

• Runbook link in annotations
• Clear summary describing what is wrong and impact
• Severity matching response expectation (page vs ticket vs log)
• Owner — which team is responsible
• Actionable — recipient can do something about it
• Tested — alert has fired in non-prod and was validated

Alert anti-patterns to avoid:

• Alerting on raw CPU% instead of SLO burn rate
• Flapping alerts (fire and resolve repeatedly)
• Alerts with no runbook ("figure it out yourself")
• Duplicate alerts for the same condition from different sources
• Alerts nobody looks at (alert fatigue → ignored pages)

Dashboard Design

Layer 1: Service Overview (start here)

One dashboard per team showing all owned services:

• Traffic (req/sec) per service
• Error rate per service (color: green <0.1%, yellow <1%, red >1%)
• p99 latency per service
• Error budget remaining per service (30-day window)

Layer 2: Per-Service Deep Dive

• RED metrics broken down by endpoint and status code
• Dependency health (upstream/downstream latencies)
• Recent deployments overlay on graphs
• Top errors table with counts and sample traces

Layer 3: Infrastructure

• USE metrics per node/pod
• Kubernetes resource utilization vs requests vs limits
• Pod restart counts, eviction counts
• Node health, disk pressure, memory pressure

Layer 4: Business Metrics

• Conversion rate, cart abandonment, signup completion
• Revenue per minute (correlated with system health)
• Feature flag adoption rates

OpenTelemetry Instrumentation

Auto-Instrumentation (Start Here)

# Node.js — auto-instrument without code changes
npm install @opentelemetry/auto-instrumentations-node
node --require @opentelemetry/auto-instrumentations-node/register app.js

# Python — auto-instrument without code changes
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
opentelemetry-instrument python app.py

Manual Spans (Add Business Context)

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('payment-service');

async function processPayment(order: Order) {
  return tracer.startActiveSpan('process-payment', async (span) => {
    span.setAttribute('order.id', order.id);
    span.setAttribute('order.amount', order.amount);
    span.setAttribute('payment.method', order.paymentMethod);
    try {
      const result = await chargeCard(order);
      span.setAttribute('payment.status', 'success');
      return result;
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

Collector Configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  # Loki ingests OTLP natively, so push logs with the core `otlphttp` exporter.
  # It appends the signal path itself: this posts to http://loki:3100/otlp/v1/logs.
  #
  # Do NOT use the old `loki` exporter here. It was deprecated on 2024-07-09 and
  # DELETED from opentelemetry-collector-contrib in v0.131.0 (2025-07-29), and
  # collector config resolution is ALL-OR-NOTHING: one unknown exporter type fails
  # the whole unmarshal at startup — `'exporters' unknown type: "loki" for id:
  # "loki"` — and the process never comes up, so the failure happens before any
  # pipeline is built. A stale logs exporter therefore takes the metrics and
  # traces pipelines down with it — you lose all three signals, not just logs.
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/loki]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]

Structured Logging

{
  "timestamp": "2026-03-21T10:30:00.000Z",
  "level": "error",
  "service": "payment-api",
  "trace_id": "abc123def456",
  "span_id": "789ghi",
  "message": "Payment processing failed",
  "error.type": "CardDeclinedException",
  "error.message": "Insufficient funds",
  "order.id": "ord-12345",
  "http.method": "POST",
  "http.path": "/api/payments",
  "http.status_code": 402,
  "duration_ms": 234
}

Rules:

• Always JSON, never unstructured text in production
• Include trace_id and span_id for correlation
• Use consistent field names (follow OTel semantic conventions)
• Redact PII: credit card numbers, passwords, tokens, email addresses
• Log levels: ERROR (action needed), WARN (degraded), INFO (significant events), DEBUG (dev only, never in prod)

Cost Management

Observability can become expensive. Control costs with:

• Sampling: Use head-based (10% of traces) or tail-based (100% of errors/slow) in OTel Collector
• Retention: 15 days for metrics, 7 days for logs, 3 days for traces (adjust per need)
• Cardinality: Never use user_id, request_id, or session_id as metric labels
• Aggregation: Pre-aggregate in recording rules, query aggregated data
• Tiering: Hot (recent, fast queries) → Warm (older, slower) → Cold (archive, S3)