Observability Setup

Observability Setup

Overview

Observability is the ability to ask arbitrary questions about a system's behavior from the outside, using its telemetry. This skill instruments a service across the three pillars — structured logs, metrics, and distributed traces — then turns raw signals into SLIs, SLOs, error budgets, and actionable alerts. The goal is not "more dashboards" but the ability to answer "is the user in pain, and where?" in under five minutes.

Keywords: observability, OpenTelemetry, OTel, structured logging, JSON logs, Prometheus, metrics, RED method, USE method, distributed tracing, spans, trace context, correlation ID, SLI, SLO, error budget, burn rate, alerting, Alertmanager, Grafana, on-call, golden signals, cardinality, exemplars.

Standardize on OpenTelemetry (OTel) for instrumentation — it is vendor-neutral and emits to Prometheus, Tempo, Loki, Datadog, Honeycomb, etc. Keep instrumentation code free of vendor SDKs; push vendor choice to the OTel Collector.

Workflow

1. Inventory the service. Identify language/runtime, request entry points (HTTP/gRPC/queue), critical user journeys, downstream dependencies, and the existing telemetry backend (or pick one). Read references/instrumentation-patterns.md for per-pillar guidance.
2. Add structured logging first. Emit one JSON event per request with a stable schema (timestamp, level, message, trace_id, span_id, service, plus event-specific fields). Never log secrets/PII. Logs become useful only once they are queryable and correlated to traces.
3. Instrument metrics using RED + USE. For request-driven services apply RED (Rate, Errors, Duration); for resources apply USE (Utilization, Saturation, Errors). Use the four golden signals (latency, traffic, errors, saturation) as the checklist. Watch label cardinality.
4. Add distributed tracing. Auto-instrument the framework, propagate W3C traceparent across service boundaries, and add manual spans around meaningful business operations. Attach trace_id to every log line and metric exemplar so you can pivot signal → signal.
5. Wire the OTel Collector. Run a collector (agent + gateway) to receive OTLP, batch, sample, and export. This decouples app code from backends and centralizes sampling/redaction. See references/instrumentation-patterns.md.
6. Define SLIs and SLOs. Pick 1–3 SLIs per user journey (availability, latency, correctness). Set realistic SLO targets and a measurement window. Use references/slo-cookbook.md and the template in templates/slo-spec.md.
7. Compute error budgets and burn-rate alerts. Alert on budget burn rate, not on raw thresholds. Use multi-window multi-burn-rate alerts to balance fast detection against false pages. Generate rules with scripts/gen_slo_alerts.py.
8. Validate and reduce noise. Run scripts/check_slo_burn.py against live metrics to sanity-check. Ensure every alert is actionable, links to a runbook, and pages a human only when users are affected. Everything else is a ticket or a dashboard.

Decision Framework

Which pillar answers which question?

Rule of thumb: alert on metrics, diagnose with traces, confirm with logs. All three must share trace_id.

What to make an SLI?

Pick SLIs that track what a user actually experiences at the service boundary, expressed as good events / valid events:

• Availability: non-5xx responses / total responses
• Latency: requests faster than threshold / total requests (threshold-based, NOT an average)
• Correctness/freshness: domain-specific (e.g., jobs completed within SLA / total jobs)

Do not SLO on CPU, memory, or queue depth — those are causes, not symptoms. They belong on USE dashboards.

Alert severity routing

Worked Example

See examples/instrument-flask-service.md for an end-to-end pass over a Python web service: JSON logging with trace correlation, OTel auto + manual spans, RED metrics, an availability + latency SLO, and the generated burn-rate alerts.

Best Practices

• One log schema, enforced. Centralize a logger that always emits JSON with required fields. Inject trace_id/span_id automatically so humans never set them by hand.
• Semantic conventions. Use OTel semantic attribute names (http.request.method, http.response.status_code, service.name, db.system) so backends and dashboards work out of the box.
• Tame cardinality. Never put unbounded values (user IDs, request IDs, full URLs, emails) in metric labels — they explode time-series count and cost. Put them in logs/trace attributes instead. Use route templates (/users/{id}), not raw paths.
• Sample traces intelligently. Head-sample low-value traffic; tail-sample to keep all errors and slow requests. Always keep 100% of error traces.
• SLOs over SLAs internally. Set internal SLOs stricter than any contractual SLA so you get warning before breaching customer commitments.
• Error budget drives prioritization. Budget remaining → ship features. Budget exhausted → freeze risky changes and burn down reliability work.
• Every alert has a runbook link. An alert with no documented response is noise. Include the dashboard URL, likely causes, and first diagnostic step.
• Symptom-based alerts, cause-based dashboards. Page on user pain (the SLO); use cause metrics (CPU, GC, pool saturation) to diagnose after the page fires.
• RED for services, USE for resources. Apply both checklists deliberately rather than dashboarding whatever the library exposes.

Common Pitfalls

• Logging averages and alerting on them. Averages hide tail latency. Use histograms and percentiles (p95/p99), and SLI as a ratio of requests under threshold.
• Alert per host / per pod. A 50-node fleet generates 50 pages for one bad deploy. Alert on the aggregate user-facing SLI.
• Threshold alerts with no budget context. error_rate > 1% pages constantly during normal blips. Burn-rate alerts only fire when the budget is genuinely at risk.
• Unbounded label cardinality. http_requests_total{user_id="..."} will OOM Prometheus. Keep label sets small and bounded.
• Traces without propagation. Forgetting to forward traceparent headers breaks the trace at every service hop, leaving disconnected single-service spans.
• Logging secrets/PII. Tokens, passwords, full card numbers, and personal data in logs are a security incident. Redact at the logger and again at the collector.
• Instrumenting everything, prioritizing nothing. 200 metrics nobody looks at is not observability. Start from user journeys and the golden signals.
• No valid events denominator. An SLI of "errors" without "total valid events" can't compute a meaningful budget. Always define both numerator and denominator, and what counts as valid.