Observability

Observability

Observability is the practice of instrumenting systems so you can understand their internal state from external outputs — logs, metrics, and traces.

When to Activate

• Adding logging to a new service or endpoint
• Setting up Prometheus metrics for a service
• Implementing distributed tracing across services
• Writing alerting rules or on-call runbooks
• Defining SLOs and error budgets for a service
• Building a Grafana dashboard for a service
• Debugging a production issue using logs, metrics, or traces

The Three Pillars

OpenTelemetry is the unifying standard across all three pillars: one SDK for logs, metrics, and traces, vendor-agnostic, with exporters to any backend (Grafana, Datadog, Honeycomb, Jaeger, etc.).

Structured Logging

Always emit logs as JSON. Human-readable plaintext is fine in development, but production logs must be machine-parseable. Never log sensitive data (PII, card numbers, passwords, tokens).

Mandatory Fields

Every log line must include:

Log Levels

Rule: use INFO in production, DEBUG only in development. WARN does not page on-call. ERROR does.

Code Examples

Python with structlog:

import structlog

logger = structlog.get_logger()

# Configure once at startup
structlog.configure(
    processors=[
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ]
)

# Usage — bind context, then log
log = logger.bind(service="payment-service", request_id=request_id)
log.info("payment.processing", order_id=order_id, amount=amount)
log.error("payment.failed", order_id=order_id, error=str(e))

TypeScript with pino:

import pino from 'pino';

const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  base: { service: 'payment-service' },
});

const requestLogger = logger.child({ requestId, traceId });
requestLogger.info({ orderId, amount }, 'payment.processing');
requestLogger.error({ orderId, err }, 'payment.failed');

Go with slog (stdlib, Go 1.21+):

import "log/slog"

logger := slog.With(
    slog.String("service", "payment-service"),
    slog.String("request_id", requestID),
)

logger.Info("payment.processing",
    slog.String("order_id", orderID),
    slog.Float64("amount", amount),
)
logger.Error("payment.failed",
    slog.String("order_id", orderID),
    slog.String("error", err.Error()),
)

Correlation ID Propagation

Correlation IDs let you follow a single request across multiple services and log entries.

• Generate request_id at the edge (API gateway or the first handler to receive the request)
• Pass it downstream via HTTP headers: X-Request-ID or X-Correlation-ID
• Include the ID in all log lines for the duration of that request
• Include the ID in error responses so users can report it to support
• In async flows (message queues, event streams): embed correlation_id in the message payload itself, not just headers

Middleware pattern (Express/Node):

import { v4 as uuidv4 } from 'uuid';

app.use((req, res, next) => {
  const requestId = req.headers['x-request-id'] as string ?? uuidv4();
  req.requestId = requestId;
  res.setHeader('X-Request-ID', requestId);
  req.log = logger.child({ requestId });
  next();
});

Middleware pattern (Python/FastAPI):

import uuid
from starlette.middleware.base import BaseHTTPMiddleware

class CorrelationIDMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        request_id = request.headers.get("x-request-id") or str(uuid.uuid4())
        with structlog.contextvars.bound_contextvars(request_id=request_id):
            response = await call_next(request)
            response.headers["x-request-id"] = request_id
            return response

Metrics with Prometheus

Prometheus is a pull-based metrics system: your service exposes a /metrics endpoint and Prometheus scrapes it on a configured interval (typically 15–30 seconds).

Metric Types

Prefer Histogram over Summary. Histograms can be aggregated across instances in PromQL; Summaries cannot.

Naming Conventions

Pattern: <namespace>_<subsystem>_<name>_<unit>

• payment_service_http_requests_total — counter; _total suffix implies count
• payment_service_db_query_duration_seconds — histogram; always include unit in the name
• payment_service_cache_hit_ratio — gauge; ratio 0–1, no unit suffix needed
• payment_service_queue_depth — gauge; current number of items in queue

Never include label values in metric names. Use labels instead:

# WRONG
http_500_errors_total

# RIGHT
http_errors_total{status="500"}

Label Cardinality Trap

Labels multiply time series. A metric with two labels each having 10 values creates 100 series. High-cardinality labels (user IDs, order IDs, full URL paths with path parameters) can create millions of series, crashing Prometheus.

# BAD — creates a new time series for every unique user ID
http_requests_total.labels(user_id=user.id).inc()

# GOOD — low cardinality labels only
http_requests_total.labels(method="POST", endpoint="/payments", status="200").inc()

Safe label values: HTTP method, endpoint (normalized, no IDs), status code, region, environment, service version.

RED Method (for services)

Use RED to define the minimum set of metrics for any service:

• Rate: requests per second

  rate(http_requests_total[5m])
  

• Errors: error rate as a fraction of total traffic

  rate(http_requests_total{status=~"5.."}[5m])
  / rate(http_requests_total[5m])
  

• Duration: latency at the 95th percentile

  histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m]))
  

USE Method (for resources)

Use USE to define metrics for infrastructure and resource-level components:

• Utilization: percentage of time the resource is busy (e.g., CPU%, disk I/O%)
• Saturation: work that is queued because it cannot be served immediately (e.g., run queue length)
• Errors: error events from the resource (e.g., disk I/O errors, network drops)

Code Examples

Python with prometheus_client:

from prometheus_client import Counter, Histogram, start_http_server

REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)
REQUEST_DURATION = Histogram(
    'request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint'],
    buckets=[.005, .01, .025, .05, .1, .25, .5, 1, 2.5]
)

# Usage
with REQUEST_DURATION.labels(method='POST', endpoint='/payments').time():
    result = process_payment()
REQUEST_COUNT.labels(method='POST', endpoint='/payments', status='200').inc()

# Expose metrics endpoint
start_http_server(8000)

TypeScript with prom-client:

import { Counter, Histogram, collectDefaultMetrics, register } from 'prom-client';

collectDefaultMetrics({ prefix: 'payment_service_' });

const requestCount = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'endpoint', 'status'],
});

const requestDuration = new Histogram({
  name: 'request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'endpoint'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5],
});

// Expose metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

Go with prometheus/client_golang:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    requestCount = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests",
    }, []string{"method", "endpoint", "status"})

    requestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "request_duration_seconds",
        Help:    "HTTP request duration",
        Buckets: prometheus.DefBuckets,
    }, []string{"method", "endpoint"})
)

// Expose metrics endpoint
http.Handle("/metrics", promhttp.Handler())

Distributed Tracing with OpenTelemetry

Concepts

• Trace: the complete end-to-end journey of a single request, identified by a unique trace_id
• Span: a single named unit of work within a trace (has a start time, end time, status, and attributes); spans form a tree
• Context propagation: the mechanism for passing trace_id and span_id across service boundaries via HTTP headers; use the W3C traceparent header (standardized, supported everywhere)
• Collector: a sidecar or standalone service (otel-collector) that receives span data, batches it, and forwards it to a backend (Jaeger, Tempo, etc.)

Auto-Instrumentation vs Manual Spans

• Auto-instrumentation: instrument HTTP frameworks, DB drivers, gRPC clients automatically — get traces with zero code changes (install the SDK + one setup file)
• Manual spans: wrap custom business logic to make it visible in traces (e.g., "processing payment", "calling fraud check API", "serializing response")

When in doubt: start with auto-instrumentation to get baseline visibility, then add manual spans where you need business context.

Setup (Python)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Bootstrap once at startup
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("payment-service")

# Manual span around business logic
with tracer.start_as_current_span("process_payment") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("payment.amount", amount)
    span.set_attribute("payment.currency", currency)
    result = charge_card(card_token, amount)
    span.set_attribute("payment.result", result.status)

Setup (TypeScript)

import { trace, SpanStatusCode, context } from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';

// Bootstrap once at startup
const provider = new NodeTracerProvider();
provider.addSpanProcessor(
  new BatchSpanProcessor(
    new OTLPTraceExporter({ url: 'http://otel-collector:4317' })
  )
);
provider.register();

const tracer = trace.getTracer('payment-service');

// Manual span with error handling
async function processPayment(orderId: string, amount: number) {
  const span = tracer.startSpan('process_payment');
  return context.with(trace.setSpan(context.active(), span), async () => {
    span.setAttribute('order.id', orderId);
    span.setAttribute('payment.amount', amount);
    try {
      const result = await chargeCard(orderId, amount);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Setup (Go)

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func initTracer(ctx context.Context) (*trace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("payment-service"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

// Usage
tracer := otel.Tracer("payment-service")
ctx, span := tracer.Start(ctx, "process_payment")
defer span.End()
span.SetAttributes(
    attribute.String("order.id", orderID),
    attribute.Float64("payment.amount", amount),
)

Sampling Strategies

Recommended production setup: parent-based at 10% for happy path + tail sampling to capture 100% of errors and traces with p99+ latency.

Alerting Design

Symptom-Based vs Cause-Based

Alert on symptoms (what users experience), not causes (what resources are doing):

Alert Design Rules

• Every alert must link to a runbook that explains diagnosis and remediation steps
• severity: critical means page someone immediately — it is happening now and users are affected
• severity: warning means investigate during business hours — degraded but not yet user-impacting
• Avoid alerting on individual metrics; alert on SLO burn rate where possible
• Every alert should be actionable: if you cannot do anything about it, do not alert on it
• Add a for: duration to avoid flapping on transient spikes (5 minutes is a safe default)

Prometheus Alerting Rule Example

groups:
  - name: payment-service
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{service="payment-service",status=~"5.."}[5m])
          / rate(http_requests_total{service="payment-service"}[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
          service: payment-service
          team: payments
        annotations:
          summary: "High error rate on payment-service"
          description: "Error rate is {{ $value | humanizePercentage }} over last 5 minutes"
          runbook: "https://wiki.example.com/runbooks/payment-service#high-error-rate"

      - alert: HighLatencyP99
        expr: |
          histogram_quantile(0.99,
            rate(request_duration_seconds_bucket{service="payment-service"}[5m])
          ) > 2.0
        for: 5m
        labels:
          severity: warning
          service: payment-service
          team: payments
        annotations:
          summary: "P99 latency above 2s on payment-service"
          description: "P99 is {{ $value | humanizeDuration }}"
          runbook: "https://wiki.example.com/runbooks/payment-service#high-latency"

SLOs and Error Budgets

Hierarchy

• SLI (Service Level Indicator): the specific metric being measured (e.g., fraction of HTTP requests returning 2xx, p99 latency)
• SLO (Service Level Objective): the internal target for an SLI (e.g., 99.9% of requests return 2xx over a rolling 30-day window)
• SLA (Service Level Agreement): the contractual commitment with defined penalties if breached; always set SLA below SLO to leave buffer

Writing a Good SLO

An SLO must specify:

1. The SLI being measured
2. The target percentage
3. The time window (rolling 30 days is standard)
4. What counts as "good" vs "bad" (the success criterion)

Examples:

• "99.9% of payment API requests return HTTP 2xx over a rolling 30-day window"
• "99% of payment API requests complete in under 500ms over a rolling 30-day window"
• "The payment service is reachable from all regions 99.95% of the time over 30 days"

Availability SLO Math

Most internal services target 99.9%. Customer-facing critical paths target 99.95%–99.99%. Do not set SLOs higher than you can actually achieve; an SLO you always exceed has no meaning.

Error Budget Policy

• Error budget = 1 − SLO (e.g., for 99.9% availability, 0.1% of requests can fail over 30 days)
• Track remaining error budget in a Grafana dashboard so teams see how much headroom they have

Burn Rate Alerting

Burn rate measures how fast you are spending your error budget relative to the expected pace.

• If SLO is 99.9% over 30 days, you have ~43.8 minutes of downtime budget per month
• A burn rate of 1.0 means you are consuming budget at exactly the sustainable rate
• A burn rate of 14x means you will exhaust your monthly budget in ~3 days

# 1-hour burn rate for a 99.9% SLO
(
  1 - (
    rate(http_requests_total{status=~"2.."}[1h])
    / rate(http_requests_total[1h])
  )
) / (1 - 0.999)

Recommended burn rate alerts:

Dashboard Design (Grafana)

Golden Signals Dashboard Template

Every service should have a single dashboard with at minimum four panels, one per golden signal:

1. Traffic — request rate (RPS) over time

   sum(rate(http_requests_total{service="$service"}[5m])) by (endpoint)
   

2. Latency — p50, p95, and p99 response time on one panel

   histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket{service="$service"}[5m])) by (le))
   histogram_quantile(0.95, sum(rate(request_duration_seconds_bucket{service="$service"}[5m])) by (le))
   histogram_quantile(0.50, sum(rate(request_duration_seconds_bucket{service="$service"}[5m])) by (le))
   

3. Errors — error rate as a percentage of total traffic

   sum(rate(http_requests_total{service="$service",status=~"5.."}[5m]))
   / sum(rate(http_requests_total{service="$service"}[5m]))
   

4. Saturation — CPU%, memory%, and connection pool usage on one panel

   avg(container_cpu_usage_seconds_total{pod=~"$service.*"}) by (pod)
   

Useful Panel Types

Dashboard Organization Tips

• Add a row per service tier (frontend, API, database, queue)
• Use template variables ($service, $env, $region) so dashboards are reusable
• Link each panel's title to the relevant runbook or alert rule
• Set the default time range to "Last 3 hours" with a 30-second auto-refresh for live debugging
• Pin the error budget remaining as a Stat panel at the top of every service dashboard

Red Flags

• High-cardinality label in a Prometheus metric — adding user_id, order_id, or an un-normalized URL path as a label creates millions of time series and can crash Prometheus under memory pressure; only use low-cardinality labels
• Alerting on CPU or memory utilization as a primary signal — CPU at 80% may have zero user impact while a queue backlog at 10% causes data loss; alert on symptoms (error rate, latency) and use resource metrics only for capacity forecasting
• Alert with no for: duration — an alert that fires on the first data point trips on transient 1-second spikes; always add for: 5m or similar to require the condition to be sustained before paging
• Logging at DEBUG level in production — debug logs from a high-traffic service generate gigabytes per hour, burying actionable signals and inflating log storage costs; configure LOG_LEVEL=INFO in production
• Generating a new request_id at each service hop — if each service creates its own ID, you cannot correlate log lines across service boundaries; generate once at the edge and propagate via X-Request-ID or the W3C traceparent header
• Using Summary instead of Histogram for latency metrics — summaries calculate quantiles client-side and cannot be aggregated across multiple instances in PromQL; histograms aggregate correctly across replicas
• SLO defined as "99.9% uptime" without specifying the SLI or success criterion — "uptime" is ambiguous; a well-formed SLO specifies the SLI (fraction of HTTP 2xx responses), the target (99.9%), and the window (rolling 30 days)
• Span attributes containing PII (email, card number, user name) — trace data is often forwarded to third-party backends (Datadog, Honeycomb) with relaxed data retention; scrub or hash sensitive fields before attaching them to spans

Checklist

• All log lines include timestamp, level, service, trace_id, span_id, and request_id
• Log level is INFO in production; DEBUG only in development; never log sensitive data
• Correlation ID is generated at the edge and propagated through all downstream services and async queues
• RED metrics (rate, errors, duration) are instrumented on every service entry point
• Prometheus metric labels have low cardinality — no user IDs, order IDs, or raw URL paths
• Histogram buckets are tuned to the expected latency range of the service
• OpenTelemetry traces are configured with an OTLP exporter pointing at a collector
• Context propagation uses the W3C traceparent header across all HTTP and gRPC calls
• Every alert has a severity label, a for: duration, and a runbook annotation
• SLOs are defined with an explicit SLI, target percentage, and rolling time window
• Error budget policy is documented: what the team does when 75%, 100% of budget is consumed
• Grafana dashboard includes all four golden signals (traffic, latency, errors, saturation) with template variables