Backend Observability

Observability Instrumentation

Degree of freedom: MIXED. What to wrap and alert on [HIGH freedom]; correlation ids, PII redaction, and the DoD [LOW freedom — run exactly].

How to reason

1. Observe — which ids, logs, spans, and PII already exist
2. Interpret — can you pivot error ↔ trace ↔ log ↔ user from any one?
3. Classify — add-correlation / convert-console / redact / wrap-span / alert
4. Severity — uncorrelated prod 500 outranks a missing debug field

Worked example

Observe: checkout 500 in Sentry has no request_id; pino logs the cart id; no LLM (Langfuse unused). Interpret: cannot pivot error → logs; webhook still console.logs. Classify: ALS request_id on logs + Sentry.setTag; convert console to pino; redact Authorization. Verify: one id on the Sentry event and the log line; beforeSend strips the header.

Self-critique before reporting

• Correlated — same request/trace id on the log, Sentry, and Langfuse
• Redacted — deny-list at the logger + beforeSend, not per-call hope
• Leveled — no console.log in prod; handled paths are not error
• Right owner — plan-only audit → plan-error-handling; investigate a Sentry issue → debug-sentry-monitor

The build-time counterpart to your monitoring stack. The Sentry plugin installs the SDK; debug-sentry-monitor triages after the fact; audit-langfuse-llm audits LLM traces. about instrumenting correctly while you build so those tools have signal to work with — and so a 3am incident is debuggable.

When this fires

Adding logging / tracing / metrics to new code, reviewing instrumentation, or fixing "we can't tell what happened in prod." Not for installing an SDK (use the Sentry/Langfuse plugins) or post-hoc triage (use the monitor/audit skills).

The one rule that matters most: correlation [LOW freedom — run exactly]

A prod incident is only debuggable if you can pivot error ↔ trace ↔ log ↔ user from any one of them. Make every layer share an id.

• Generate/propagate a request id (or OTel trace_id) at the entry point (HTTP middleware, edge function, job start). Put it in async context (AsyncLocalStorage / context var), not a parameter threaded everywhere.
• Stamp it on everything: every log line, Sentry setTag("request_id", id) / setContext, and the Langfuse trace (trace.id or metadata). On an LLM error, attach the Langfuse trace URL to the Sentry event so you jump straight from the error to the prompt/response.
• Set user/session/tenant scope (Sentry.setUser, Langfuse userId/sessionId) — scrubbed (see redaction).

Structured logging discipline [HIGH freedom]

• Structured, not string-soup. Emit JSON with stable fields (level, msg, request_id, event, domain ids). One event per line. Use the platform logger (pino / structlog / slog), not bare console.log.
• Levels mean things: error = needs a human; warn = degraded but handled; info = state transitions / business events; debug = dev-only, off in prod. Don't log error for handled flow.
• No console.log shipped to prod — it's unsearchable, unleveled, and a PII leak risk. Remove or convert to a leveled structured log.
• Log decisions, not noise: log the branch taken + key inputs/outputs at boundaries, not every line. A log you'd never grep is cost, not signal.

PII / secret redaction (non-negotiable) [LOW freedom — run exactly]

• Never log tokens, passwords, API keys, full PANs, auth headers, or raw request bodies. Redact at the logger/transport layer (deny-list keys + pattern scrub) so it can't be bypassed per-call.
• Configure Sentry beforeSend / data-scrubbing and Langfuse masking to strip PII from events/traces. Assume anything you put in a span/breadcrumb may be retained.
• For LLM traces: decide explicitly whether prompts/completions may contain PII; mask or hash before sending if the jurisdiction requires it.

Tracing / spans (what to wrap) [HIGH freedom]

• Wrap boundaries and slow/fallible work: inbound request, outbound HTTP/DB/queue calls, LLM calls, background jobs. Not every function.
• Name spans by operation (http.server, db.query, llm.generate), add attributes (route, status, row count, model) — follow OTel semantic conventions, including the GenAI conventions for LLM spans (model, tokens in/out, cost, latency, temperature).
• Sampling: you don't need 100%. Head-sample normal traffic (e.g. 10–20%), but always keep errors and slow outliers. Document the rate; it's a cost/visibility dial.

LLM-specific (Langfuse) [HIGH freedom]

• Capture per generation: prompt, response, model, input/output tokens, cost, latency, and the eval/score if you run one. Group multi-step agents under one trace with nested spans.
• Link the Langfuse trace_id into the surrounding request id and into Sentry on failure — so an LLM error in Sentry is one click from the full trace.
• Tag traces with userId / sessionId / release so audit-langfuse-llm can slice quality by cohort.

Alerts & SLOs (signal, not noise) [HIGH freedom]

• Alert on symptoms users feel (error-rate spike, p95 latency, checkout/login failure, LLM eval-score drop), not every error. A pager that cries wolf gets muted.
• Define a few SLOs (availability, latency, key-flow success) and alert on burn rate, not raw counts.
• Every alert names an owner and a first action. Route via sentry-create-alert / your channel.

Definition of done [LOW freedom — do not skip]

• A shared request/trace id is on every log line, the Sentry scope, and the Langfuse trace.
• From a prod error you can reach the trace, the logs, and the user in ≤2 clicks.
• Logs are structured + correctly leveled; no console.log shipped to prod.
• PII/secret redaction is enforced at the logger + Sentry beforeSend + Langfuse masking.
• Spans cover boundaries with OTel-conventional names/attributes; errors are never sampled out.
• LLM generations record model/tokens/cost/latency and link back to the request id.
• Alerts fire on user-felt symptoms with an owner, not on every error.

Composes with

• Sentry plugin (sentry-sdk-setup, sentry-setup-ai-monitoring, sentry-create-alert, sentry-otel-exporter-setup) — SDK + alert wiring.
• Langfuse plugin (langfuse) + audit-langfuse-llm — LLM trace capture + quality audit.
• debug-sentry-monitor / debug-error — post-hoc triage of what this instrumentation surfaces.
• data-pipeline — per-run pipeline metrics use these same correlation + logging rules.
• workflow-spec-tdd — make "observable" part of the spec's "done when", not an afterthought.