Backend Observability

backend-observability

You cannot debug what you cannot see. This skill instruments a service so that when it misbehaves at 3am, an on-call engineer can answer what broke, for whom, and where from telemetry alone — without redeploying to add a log line. The three pillars (logs, metrics, traces) are wired to a single correlation id so you can pivot between them.

Use when

• A service emits console.log/print and nothing else.
• Incidents take too long to diagnose because there's no signal.
• As the observability/ step when scaffolding a new service.

The one thread: correlation id

Every request gets a correlation id at the edge (accept an inbound traceparent/X-Request-Id if present, else generate one). It flows through logs (as a field), traces (as the trace id), and to downstream calls (as a propagated header). This is what lets you take a log line, jump to its trace, and see the metric it contributed to. Wire this first — the pillars are far less useful uncorrelated.

Structured logs

• JSON, one object per line. Never interpolate values into a message string you'll later have to regex. log.info("order placed", { orderId, userId, amountCents }), not log.info("order " + id + " placed").
• Levels mean things. error = a human should look; warn = degraded but handled; info = business milestones; debug = off in prod. If everything is error, nothing is.
• Every log line carries the correlation id, service name, and version. Add request-scoped context (route, user id) via a context/AsyncLocalStorage so you set it once per request.
• Log at boundaries, not everywhere. One line when a request enters, one when it leaves (with status + duration), one per external call. Not a line per function.

Metrics — RED for services, USE for resources

Instrument request-handling with RED: Rate (requests/sec), Errors (failed/sec), Duration (latency distribution). Instrument resources (pools, queues, caches) with USE: Utilization, Saturation, Errors.

• Duration is a histogram, never an average — you need p50/p95/p99. An average hides the tail where the pain lives.
• Errors are a counter you can divide by rate to get an error ratio — alert on the ratio, not the raw count (raw count scales with traffic).
• Emit business metrics too where they're cheap: orders placed, signups, payments failed. These catch outages that leave the infra green.

Cardinality is the footgun

Every unique combination of label values is a separate time series. High-cardinality labels (user id, request id, email, full URL path with ids) multiply series into the millions and take down your metrics backend and your bill.

Rule: if you can't enumerate a label's possible values in advance and they're few, it doesn't belong on a metric.

Distributed tracing

• Use OpenTelemetry — vendor-neutral, exports to Jaeger/Tempo/Datadog/etc. Don't hand-roll spans against one vendor's SDK.
• Auto-instrument the framework + HTTP client + DB driver for free spans; add manual spans only around meaningful business operations.
• Propagate context across service boundaries (W3C traceparent) and across async boundaries (queues: inject trace context into the message, extract on consume) — a trace that stops at the queue is half a story.
• Sample. Head-based sampling at a sane rate (e.g. 10%) for volume, but always sample errors and slow requests (tail-based if the backend supports it). You want every failure traced, not 10% of them.

PII and secrets — do not log them

• Never log passwords, tokens, full card numbers, or auth headers. Redact at the logger, not by remembering to omit them at each call site — build a redaction serializer that masks known-sensitive keys (authorization, password, token, ssn, …).
• Be deliberate about emails/user ids in logs vs metrics: fine in a log (access-controlled, retained short), never as a metric label (see cardinality).
• Assume logs and traces may be shipped to a third-party backend — that's a data-egress decision, treat it like one.

Procedure

1. Wire the correlation id at the edge + propagation to downstream calls and async messages.
2. Replace ad-hoc prints with a structured logger; add the redaction serializer; set request-scoped context.
3. Add RED metrics on the request path (histogram for duration) and USE metrics on pools/queues; audit labels against the cardinality table.
4. Add OpenTelemetry tracing: auto-instrument, propagate context, sample with errors-always.
5. Add the three or four alerts that matter (below), not fifty.
6. Verify by causing a failure in staging and diagnosing it from telemetry only — if you can't, the instrumentation isn't done.

Alerts worth having (start here, not with fifty)

• Error ratio over threshold, sustained (page).
• p99 latency over SLO, sustained (page if user-facing).
• Saturation of a critical resource (pool exhausted, queue backing up) (page).
• A key business metric flatlining (e.g. zero orders in N minutes) (page). Everything else is a dashboard, not a page. Alert fatigue is an outage waiting to happen.

Definition of done

• One correlation id joins a log line → its trace → the downstream call.
• Logs are structured JSON with levels used meaningfully and sensitive keys redacted.
• RED + USE metrics exist; no unbounded metric labels; duration is a histogram.
• Traces propagate across service and queue boundaries; errors always sampled.
• A synthetic failure in staging is diagnosable from telemetry without adding code.