# Backend Observability

> Instrument a backend service so failures are diagnosable in production — structured logs, RED/USE metrics, and distributed tracing wired through one correlation id. Use when a service has no observability, when incidents take too long to diagnose, or as the observability step of api-service-scaffold. Vendor-neutral (OpenTelemetry), with concrete cardinality and PII guardrails.

- Skill: `omonuj/backend-observability` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/backend-observability`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/backend-observability/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/backend-observability

---


# backend-observability

You cannot debug what you cannot see. This skill instruments a service so that when it misbehaves at 3am, an on-call engineer can answer *what broke, for whom, and where* from telemetry alone — without redeploying to add a log line. The three pillars (logs, metrics, traces) are wired to a single correlation id so you can pivot between them.

## Use when
- A service emits `console.log`/`print` and nothing else.
- Incidents take too long to diagnose because there's no signal.
- As the `observability/` step when scaffolding a new service.

## The one thread: correlation id
Every request gets a correlation id at the edge (accept an inbound `traceparent`/`X-Request-Id` if present, else generate one). It flows through logs (as a field), traces (as the trace id), and to downstream calls (as a propagated header). This is what lets you take a log line, jump to its trace, and see the metric it contributed to. Wire this first — the pillars are far less useful uncorrelated.

## Structured logs
- **JSON, one object per line.** Never interpolate values into a message string you'll later have to regex. `log.info("order placed", { orderId, userId, amountCents })`, not `log.info("order " + id + " placed")`.
- **Levels mean things.** `error` = a human should look; `warn` = degraded but handled; `info` = business milestones; `debug` = off in prod. If everything is `error`, nothing is.
- **Every log line carries** the correlation id, service name, and version. Add request-scoped context (route, user id) via a context/AsyncLocalStorage so you set it once per request.
- **Log at boundaries, not everywhere.** One line when a request enters, one when it leaves (with status + duration), one per external call. Not a line per function.

## Metrics — RED for services, USE for resources
Instrument request-handling with **RED**: **R**ate (requests/sec), **E**rrors (failed/sec), **D**uration (latency distribution). Instrument resources (pools, queues, caches) with **USE**: **U**tilization, **S**aturation, **E**rrors.

- Duration is a **histogram**, never an average — you need p50/p95/p99. An average hides the tail where the pain lives.
- Errors are a counter you can divide by rate to get an error *ratio* — alert on the ratio, not the raw count (raw count scales with traffic).
- Emit business metrics too where they're cheap: orders placed, signups, payments failed. These catch outages that leave the infra green.

### Cardinality is the footgun
Every unique combination of label values is a separate time series. High-cardinality labels (user id, request id, email, full URL path with ids) multiply series into the millions and take down your metrics backend and your bill.

| Label | OK as metric label? |
| --- | --- |
| `route` (templated: `/users/:id`) | ✅ bounded |
| `method`, `status_code` | ✅ bounded |
| `user_id`, `order_id`, `email` | ❌ unbounded — put it in the **trace/log**, never a metric label |
| raw `path` (`/users/8134`) | ❌ unbounded — template it first |
| `region`, `tenant` (if bounded) | ⚠️ OK only if genuinely low-cardinality |

Rule: if you can't enumerate a label's possible values in advance and they're few, it doesn't belong on a metric.

## Distributed tracing
- Use **OpenTelemetry** — vendor-neutral, exports to Jaeger/Tempo/Datadog/etc. Don't hand-roll spans against one vendor's SDK.
- Auto-instrument the framework + HTTP client + DB driver for free spans; add manual spans only around meaningful business operations.
- Propagate context across service boundaries (W3C `traceparent`) and across async boundaries (queues: inject trace context into the message, extract on consume) — a trace that stops at the queue is half a story.
- **Sample.** Head-based sampling at a sane rate (e.g. 10%) for volume, but always sample errors and slow requests (tail-based if the backend supports it). You want every failure traced, not 10% of them.

## PII and secrets — do not log them
- Never log passwords, tokens, full card numbers, or auth headers. Redact at the logger, not by remembering to omit them at each call site — build a redaction serializer that masks known-sensitive keys (`authorization`, `password`, `token`, `ssn`, …).
- Be deliberate about emails/user ids in logs vs metrics: fine in a log (access-controlled, retained short), never as a metric label (see cardinality).
- Assume logs and traces may be shipped to a third-party backend — that's a data-egress decision, treat it like one.

## Procedure
1. Wire the correlation id at the edge + propagation to downstream calls and async messages.
2. Replace ad-hoc prints with a structured logger; add the redaction serializer; set request-scoped context.
3. Add RED metrics on the request path (histogram for duration) and USE metrics on pools/queues; audit labels against the cardinality table.
4. Add OpenTelemetry tracing: auto-instrument, propagate context, sample with errors-always.
5. Add the three or four alerts that matter (below), not fifty.
6. Verify by causing a failure in staging and diagnosing it *from telemetry only* — if you can't, the instrumentation isn't done.

## Alerts worth having (start here, not with fifty)
- Error ratio over threshold, sustained (page).
- p99 latency over SLO, sustained (page if user-facing).
- Saturation of a critical resource (pool exhausted, queue backing up) (page).
- A key business metric flatlining (e.g. zero orders in N minutes) (page).
Everything else is a dashboard, not a page. Alert fatigue is an outage waiting to happen.

## Definition of done
- One correlation id joins a log line → its trace → the downstream call.
- Logs are structured JSON with levels used meaningfully and sensitive keys redacted.
- RED + USE metrics exist; no unbounded metric labels; duration is a histogram.
- Traces propagate across service and queue boundaries; errors always sampled.
- A synthetic failure in staging is diagnosable from telemetry without adding code.