# Observability Setup

> Instrument a backend with the three signals unified by one correlation context — structured logs, metrics (RED for services, USE for resources), and distributed tracing (OpenTelemetry + W3C Trace Context). Use before production, when debugging is blind, or when an incident has no trail. Not for diagnosing specific bottlenecks (use performance-profiling) or AI-specific token/cost metrics (use ai-llm-backend on top of this backbone).

- Skill: `jaykim88/observability-setup` (Agent Skill)
- Install (CLI): `npx skillmds@latest add jaykim88/observability-setup`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jaykim88/observability-setup/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: JayKim88 (https://skillmd.com/u/jaykim88)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/jaykim88/observability-setup

---


# Observability Setup (Backend)

## Purpose
Turn production silence into signal by emitting three correlated signals — logs, metrics, traces — joined by one propagated context, so any request can be reconstructed end-to-end and any regression is visible.

**Universal** — the three-signals model, RED/USE methods, and trace-context propagation are vendor-neutral (OpenTelemetry is the CNCF standard); implemented across Node/Python/Go.

## Procedure

1. **Propagate ONE correlation context**
   - Generate/accept a `traceId` (W3C Trace Context header) at the edge
   - Include it in every log line, span, and downstream call (incl. background jobs + queue messages)
   - This is what lets logs/metrics/traces *join* — without it you have three disconnected silos

2. **Structured logging (JSON)**
   - `{ level, timestamp, traceId, route, userId(anonymized), msg, ...context }`
   - No `console.log`; one logger utility; levels (debug/info/warn/error)
   - Never log secrets/PII

3. **Metrics — RED for services, USE for resources**
   - **RED** (services/endpoints): Rate, Errors, Duration (p50/p95/p99)
   - **USE** (resources: DB pool, CPU, queue): Utilization, Saturation, Errors
   - Apply RED to request handlers, USE to infrastructure — different questions, different methods
   - **Cardinality budget**: every unique label combination is a stored time series — putting `userId`, `requestId`, `email`, or unbounded ids in labels explodes the TSDB. Keep labels low-cardinality (route, status code, region); push per-request detail into logs/traces, not metrics

4. **Distributed tracing (OpenTelemetry) — and sample it**
   - Auto-instrument the framework + DB + HTTP client; add manual spans around key business operations
   - Spans carry the traceId; a trace shows the full request waterfall (where time went)
   - **Sampling**: 100% trace capture is unaffordable at scale. Choose: head-based (decide at request start, simplest), tail-based (decide after the trace completes — keeps all errors + slow traces, drops the boring middle), or adaptive. Default to head-based at a low rate + always-on for errors

5. **Capture errors with context — and scrub PII**
   - Errors → error tracker (Sentry) WITH traceId + anonymized user context
   - Any `catch` that returns gracefully must still record the error (silent catch = invisible failure)
   - The SDK captures *more than you set*: request URLs, headers, breadcrumbs (and span attributes in tracing) can leak tokens, auth headers, and PII. Configure `beforeSend` scrubbing + platform-side data scrubbing; treat the trace/log/error pipeline as the same trust boundary as your API responses

6. **Alert on patterns, not noise**
   - Alert on RED/USE threshold breaches (error rate spike, p99 regression, queue saturation), not every error
   - Define SLOs; alert on burn rate

7. **Validate (validation loop)**
   - Trigger a test error + a slow request → verify you can find them by traceId across logs + trace + error tracker
   - If the three signals don't join on traceId → context propagation is broken; fix and re-test

## Anti-patterns

| ❌ Anti-pattern | ✅ Correct |
|---|---|
| `console.log` everywhere | Structured JSON logger with traceId |
| Logs/metrics/traces with no shared id | One propagated W3C trace context |
| Averages only | p95/p99 (tail latency is what users feel) |
| RED applied to infra / USE to endpoints | RED→services, USE→resources |
| Alert on every error | Alert on SLO burn / threshold breach |
| Silent catch (no record) | Record every handled error with context |
| `userId` / `requestId` / `email` in metric labels (TSDB explosion) | Low-cardinality labels (route, status, region); per-request detail in logs/traces |
| 100 % trace capture in production | Head- or tail-based sampling; always-on for errors + slow traces |
| Trusting SDK auto-captured payloads to be PII-free | `beforeSend` scrubbing + platform data scrubbing; treat the pipeline as a trust boundary |

## Severity tiers

| Tier | Examples | Action SLA |
|---|---|---|
| **Critical** | No error tracking in production; silent catches hiding failures; no correlation id (can't trace incidents) | Block release; fix immediately |
| **Major** | Metrics on averages only (no p99); traces not propagated to jobs/queues; PII / auth headers leaking into traces or breadcrumbs (no `beforeSend` scrubbing) | Fix this sprint |
| **Minor** | Alert noise (per-error alerts); missing USE metrics on a resource; metric cardinality unbudgeted (label explosion risk); trace sampling rate unset | Schedule within 2 sprints |

## Completion Criteria
- [ ] One correlation context propagated through requests + jobs + queues
- [ ] Structured JSON logging (no console.log; traceId on every line)
- [ ] RED metrics on services, USE on resources, p95/p99 tracked
- [ ] OpenTelemetry tracing wired (auto + key manual spans)
- [ ] Errors captured with traceId; SLO-based alerts
- [ ] Trace sampling rate set (head- or tail-based); always-on for errors / slow traces
- [ ] Metric labels low-cardinality (no userId / unbounded ids); a cardinality budget documented
- [ ] PII auto-capture scrubbed (`beforeSend` + platform data scrubbing)
- [ ] Three signals verified to join on traceId

## Output
- **Instrumentation**: OTel setup + structured logger + metrics
- **Dashboards**: RED + USE
- **Alert config**: `docs/observability-alerts.md` — SLOs, thresholds, runbooks
- **Commit format**: `feat(obs): wire OpenTelemetry tracing` / `feat(obs): RED metrics for <service>`

## Implementation

### TypeScript + NestJS (default)
- Tracing: `@opentelemetry/sdk-node` + auto-instrumentations (HTTP, Prisma, Redis, BullMQ); export to Tempo/Jaeger/Datadog
- Logging: `pino` (JSON) with a traceId field from the active span context
- Metrics: `prom-client` exposing `/metrics` for Prometheus; Grafana dashboards
- Errors: `@sentry/node` with `tracesSampleRate`; attach traceId
- Propagate traceId into BullMQ job data

### Other stacks
- **Python / FastAPI**: `opentelemetry-instrumentation-fastapi`; `structlog`; `prometheus_client`
- **Go**: `go.opentelemetry.io/otel`; `slog` (stdlib structured logging); `prometheus/client_golang`
- **Universal**: OpenTelemetry + W3C Trace Context are the cross-language standard; RED/USE are methodologies, not tools

## Related skills
- `performance-profiling` — traces/metrics surface the bottlenecks profiling then drills into
- `background-jobs` — jobs need the same correlation context as requests
- `ai-llm-backend` — token/cost/latency are AI-specific metrics on this backbone

## Reference
- **Key insight encoded**: Propagate one correlation context (traceId in every log + W3C Trace Context header) so logs/metrics/traces join; apply RED to services, USE to infrastructure. Three cost-and-safety gates often missed: trace sampling (100% is unaffordable at scale), a metric-cardinality budget (one bad label explodes the TSDB), and SDK-level PII scrubbing — the pipeline auto-captures more than you set.