Observability

Observability and Instrumentation

Code you can't observe is code you can't operate. Instrumentation is written alongside the feature, the same way tests are — if a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.

When to use

• Building a feature that will run in production (new service, endpoint, background job, external integration).
• A production incident took too long to diagnose because the system couldn't tell what happened.
• Adding or reviewing alerting rules.
• Reviewing a PR that adds I/O, retries, queues, or cross-service calls.

Not for: Diagnosing a live failure (use debugging) or profiling measured slowness (use performance). Observability is what makes those skills fast next time.

Steps

1. Define "working" before instrumenting

Write down 2–4 questions an on-call engineer will ask about this feature. Every signal you add must answer one of them — if you can't name the questions, you'll log everything and learn nothing.

If the service has no SLO, define one before setting alert thresholds. An SLO turns "is it healthy?" from a feeling into a number — see references/slo-methodology.md for SLI definition, error budgets, and burn-rate alerting.

2. Pick the right signal per question

• Structured log — "what happened in this specific case?" (per-event; grows with traffic)
• Metric — "how often / how fast, in aggregate?" (fixed per series; cheap to query)
• Trace — "where did time go across services?" (per-request; usually sampled)

Rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.

3. Structured logging

Log events, not prose. Every line is a JSON object with a stable event name and machine-readable fields.

// BAD: string interpolation — unqueryable, inconsistent
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

// GOOD: stable event name + structured fields
logger.warn({ event: 'payment_failed', paymentId: id, provider: 'stripe',
              errorCode: err.code, attempt: n }, 'payment failed');

Log levels: error (invariant broken, someone may act), warn (degraded but handled), info (significant business event), debug (off in production).

Correlation IDs are mandatory. Generate or accept a request ID at the system boundary and attach it to every log line, span, and outbound call — without it a single request can't be reconstructed from interleaved logs.

Never log secrets, tokens, passwords, or full PII. Telemetry pipelines are a classic data-leak path. Allowlist fields; don't log whole request bodies.

4. Metrics

For request-driven services instrument RED on every endpoint and external dependency: Rate, Errors, Duration (latency histogram, not average). For resources (queues, pools, hosts) use USE: Utilization, Saturation, Errors.

Cardinality is the failure mode. Every unique label combination is a separate time series. Labels must come from small, fixed sets (route template, status class, provider name). Never use user IDs, raw URLs, or error message text as labels — those belong in logs and traces.

Track averages never, percentiles always: p50/p95/p99 via histograms.

5. Distributed tracing

Use OpenTelemetry (vendor-neutral; auto-instrumentation covers HTTP, gRPC, common DB clients with near-zero code). Initialize before other imports. Add manual spans only around meaningful internal units of work (e.g. applyDiscounts, chargeProvider) with the attributes on-call will filter by. Propagate context across every async boundary — HTTP headers, queue message metadata — or the trace dies at the gap. Sample head-based at a low rate by default; keep 100% of errors if tail sampling is available.

6. Alerting

Alert on symptoms users feel (error rate, p99 latency, queue age), not on causes (CPU, disk, restarts) — cause-based alerts fire when nothing is wrong and miss failures you didn't predict.

Every alert must:

1. Be actionable — if the response is "ignore it, it self-heals", delete it.
2. Link to a runbook written when you create the alert, before any incident (minimum three lines: what it means, first query, escalation path). See ../incident-response/references/runbook-template.md for the authoring structure — the alert author owns the runbook.
3. Have a threshold and duration justified by the SLO or historical data, not a guess.
4. Use one of two severities: page (user-facing, act now) or ticket (degradation, act this week).

7. Verify the telemetry itself

Instrumentation is code; it can be wrong. Trigger the paths and look at the actual output: force an error in staging and find it by correlation ID; send test traffic and confirm metric series appear with expected labels; follow one request end-to-end in the tracing UI with no broken spans; fire each new alert once (lower the threshold temporarily) and confirm it reaches the right channel with a working runbook link.

Verify

• On-call questions for the feature are written down; each signal maps to one.
• All log output is structured JSON with stable event names and a correlation ID on every line.
• No secrets, tokens, or unredacted PII in any log line (spot-check actual output).
• RED metrics exist for every new endpoint and external dependency, with bounded label sets.
• Latency is a histogram; p95/p99 are queryable.
• A single request can be followed end-to-end in the tracing UI without broken spans.
• Every new alert is symptom-based, has a runbook link, and was test-fired once.
• An induced failure in staging was located via telemetry alone, without reading the source.

Red flags: a feature PR with retries, queues, or external calls and zero new telemetry; log lines built by string interpolation; no correlation ID on each line; metrics labeled with user IDs, raw URLs, or error text; latency tracked as an average with no percentiles; alerts firing daily and acknowledged without action; secrets or full request bodies in logs.

References

• ${CLAUDE_PLUGIN_ROOT}/references/engineering-principles.md — shared discipline (verify don't assume, enforce simplicity, surgical scope)
• references/observability-checklist.md — at-a-glance instrumentation checklist + pre-launch gate
• references/slo-methodology.md — SLI definition, error budgets, burn-rate alerting, SLO-based release gating
• references/cost-observability.md — FinOps: cost attribution, per-request cost, anomaly detection, optimization levers
• ../incident-response/references/runbook-template.md — runbook authoring structure (write when you create the alert, not during the incident)
• references/slo-implementation.md — Prometheus recording/alerting rules, multi-window burn-rate alerts, error-budget queries, dashboards, review cadence