Observability

Observability

Instrument for the questions you haven't asked yet. Monitoring verifies failure modes you predicted; observability lets you interrogate the system about failures you never anticipated. Effective telemetry answers two questions — what's broken (symptom) and why (cause) — and the effort split is lopsided: spend far more on catching symptoms than on enumerating causes (Google SRE, Monitoring Distributed Systems).

This skill covers what goes into telemetry and how it is consumed. The twelve-factor skill owns log transport and shape (structured records on platform-captured process streams, levels, timestamps). The hexagonal-architecture skill owns where instrumentation code lives in ports-and-adapters codebases.

Deep-dive resources are in the resources/ directory. Load them on demand:

When to Use

• Instrumenting a new or existing service (backend, worker, API)
• Defining SLIs/SLOs or an error budget policy
• Designing, reviewing, or pruning alerts
• "We can't see what production is doing" — debugging unknown-unknowns
• Reviewing telemetry spend, metric cardinality, or sampling strategy

Not this skill: diagnosing a failing CI run (ci-debugging), local-vs-production drift (production-parity-skill-builder), Core Web Vitals optimization (core-web-vitals where installed).

The Wide Event (Canonical Log Line)

The opinionated default: emit one structured, information-dense event per request per service. This is Stripe's canonical log line — "one long log line at the end that includes many of their key characteristics" (Stripe) — and the "arbitrarily-wide structured event" at the heart of modern observability (Charity Majors).

The mechanics:

1. Middleware creates a request-scoped accumulator when the request enters the service
2. Business logic and middleware add fields as work happens
3. The event is emitted once, at the end of the request, in finally/teardown logic — it must survive the exception path, because that is exactly when you need it

Construction rule — one exit, proved by a throw. Open the try before anything that can throw: request parsing and body reads included, not just the business call. The only call site that finishes the event — setting the outcome attributes, recording the exception, ending the span or emitting the record — sits in the matching finally. A helper that does all three is fine, but it is called from the finally and from nowhere else. A helper invoked on the success path, paired with a finally that merely ends a span or flushes, still loses every field on the throw. Before calling the work done, run a test that makes the handler throw outside its own catch and assert that one event carrying the outcome is still recorded.

What goes in it:

High-cardinality identifiers belong in events, subject to privacy policy — request IDs, build IDs, and privacy-approved or pseudonymous subject identifiers can make an event useful. High cardinality is a feature for events and a cost explosion on metrics, but event suitability is never permission to emit personal data (see Sampling and Cost Economics and Telemetry Hygiene below).

Why this beats scattered log lines: the data arrives pre-joined. You query complete rows instead of reconstructing a request from fragments with regex and hope. An OpenTelemetry root span with rich attributes is a valid implementation of the same pattern — instrumenting via spans satisfies both the wide-event and the tracing camps at once.

Pillars, Honestly

The industry framing is "three pillars": logs, metrics, traces. The serious critique (Majors, Observability 1.0 vs 2.0): three pillars means many sources of truth scattered across tools, each request stored several times, engineers correlating by hand, and cost multiplied per pillar. The alternative: one source of truth — wide structured events — from which metrics, traces, and SLOs are derived at read time.

House position:

• Wide events are the instrumentation default. They avoid duplicate instrumentation and preserve more future questions, but export, processing, and storage still need an explicit telemetry cost budget.
• Metrics still earn their place for cheap, long-retention aggregates and alerting math — with strictly bounded label sets.
• Traces are wide events with structure — parent/child causality across services. Instrument via OTel spans and you get both.

Honest limit: deriving everything at read time assumes a backend that can aggregate over raw events at scale. Not every stack has one. Don't cargo-cult either camp — emit wide events regardless (the data is the asset; backends change), and keep a small set of bounded metrics for dashboards and alert rules.

OpenTelemetry: The Substrate

OpenTelemetry (OTel) is the vendor-neutral standard for producing telemetry. Adopting it means the instrumentation outlives any backend choice.

The pieces:

• Signals — traces, metrics, logs, all exported over OTLP
• Resource attributes — service.name is mandatory; it identifies the emitting service in every backend
• Semantic conventions — standardized attribute names (http.request.method, db.system) that make telemetry correlatable across teams and tools (semconv). Never invent an attribute name semconv already defines.
• Context propagation — the W3C traceparent header carries trace ID and parent span ID across every service hop, which is what makes distributed traces exist at all (context propagation). Whenever your code builds an outbound request to another service — through a vendor SDK, a generated client, or hand-written headers — inject the active context into that request's headers (propagation.inject(context.active(), headers)) instead of writing a correlation header of your own. A downstream team asking for "something we can search on to line your call up with ours" is asking for traceparent, whether or not they know the name: x-trace-id, x-correlation-id and a bare trace id under any custom key drop the parent span id and the sampling flag, and no standard tooling joins them to anything
• The Collector — a receive → process → export pipeline that runs beside your services. It is the recommended production default for batching, retry, redaction, and backend swaps without code changes. A documented direct-export design is acceptable when the deployment is simple and the SDK/backend path demonstrably meets its buffering, retry, backpressure, filtering, and operational requirements (Collector)

Minimal TypeScript adoption: @opentelemetry/sdk-node + @opentelemetry/api + @opentelemetry/auto-instrumentations-node, an instrumentation file loaded before application code via node --import ./instrumentation.mjs (or a reviewed, repository-installed TypeScript runner during development), OTLP exporters. Auto-instrumentation gives HTTP/framework/DB spans for free; manual attributes add the business meaning that makes traces queryable. Initialization order matters — instrumentation loaded after the app's modules silently captures nothing. See resources/node-patterns.md.

Sampling and Cost Economics

The cardinality routing rule

Every unique label combination on a metric is a separate time series. Add tenant_id (10,000 values) to a metric with 1,000 existing series and you have 10 million series, not 11,000; each active series costs memory and most vendors bill per series or per custom metric (Grafana on high cardinality).

The rule: bounded, low-cardinality dimensions go on metrics; unbounded, high-cardinality dimensions go on events/spans. Request-derived labels are safe only after normalization to a fixed allowlist (for example, a known method or result category); map unknown values to _OTHER or omit them. Never use raw or unbounded attacker-controlled values such as IDs, paths, query strings, arbitrary headers, or body fields. When someone asks "can we break this metric down by customer?", the answer is "that question belongs to the event store."

The sampling ladder

(OTel sampling concepts)

Never silently sample the stream your SLOs are computed from. If sampling is unavoidable there, account for it in the SLI math. Configure and verify error-trace retention separately; tail sampling policy and capacity determine what is actually retained.

SLIs, SLOs, Error Budgets

Definitions from the Google SRE book: an SLI is a carefully defined quantitative measure of service level; an SLO is a target value for that SLI; an SLA is an SLO plus consequences. The error budget is 100% minus the SLO — and its purpose is to license shipping: spend the budget on releases instead of chasing a 100% that users can't distinguish anyway.

SLI menus (mnemonics for shopping, not mandates):

• RED — Rate, Errors, Duration — per request-driven service; explicitly a proxy for user experience (Tom Wilkie)
• USE — Utilization, Saturation, Errors — per hardware resource; infrastructure-focused (Brendan Gregg)
• Four Golden Signals — latency, traffic, errors, saturation — with two nuances everyone drops: track latency of failed requests separately (a slow 500 is a different pathology from a fast 500), and use histograms because averages hide the tail — at 1,000 rps averaging 100ms, 1% of requests can easily take 5s (SRE book)

SLO discipline: keep few SLOs, each simple enough to explain in a sentence. Define availability, correctness, and freshness as good events over valid events. Define latency objectives as the proportion of valid events below an explicit threshold; use percentiles rather than means when diagnosing latency distributions. Keep internal targets slightly stricter than published ones. Don't overachieve — users come to depend on the reliability you deliver, not the one you promised.

Alerting: Symptoms, Pages, Burn Rates

Page on symptoms, not causes. "Do your users care if your MySQL servers are down? No, they care if their queries are failing" (Rob Ewaschuk, My Philosophy on Alerting — the doc that became SRE book chapter 6). Cause-based data belongs on dashboards and in tickets, not pages. The rare exception: imminent, definite causes (quota exhaustion in 4 hours).

Every page must be: urgent, actionable, user-visible, and require human intelligence to handle. If the response to a page could be scripted, script it and delete the page. Measure false pages, then fix or remove alerts that repeatedly waste responders' attention.

The default alert construction is the multiwindow, multi-burn-rate alert (SRE Workbook). Burn rate = how fast you consume error budget relative to the SLO (burn rate 1 = exactly out of budget at period end). For a 99.9% SLO over 30 days:

The short window (1/12 of the long) confirms the problem is still happening, which fixes the reset-time failure of naive threshold alerts. This is the only construction that scores well on precision, recall, detection time, and reset time simultaneously — the derivation is in resources/slo-alerting.md.

Every page links a runbook — a concise "what this alert means and current mitigations", not an exhaustive troubleshooting tree.

Structured Logging Craft

The twelve-factor skill owns transport and shape (structured records on platform-captured process streams, recognized levels, ISO timestamps). This section is about content discipline.

The levels test (from Dave Cheney): "nobody reads warnings, because by definition nothing went wrong," and "if you choose to handle the error by logging it, by definition it's not an error any more." Keep the standard four levels — but apply Cheney's test to every line: every warn needs a named reader; every error log must correspond to a genuinely unhandled failure, not a handled one being double-reported.

Log at boundaries; accumulate in between. Most in-request info chatter should become fields on the wide event, not separate lines. A request that produces 40 log lines produces one canonical event plus a handful of genuinely independent facts.

Correlation: every log record carries the active trace ID (W3C trace context), so logs join to traces and to the canonical event for free.

PII and secret hygiene:

• Never emit passwords, tokens, API keys, cookies, or session IDs into any signal
• Emit personal identifiers only when a stated operational need and the applicable privacy policy permit it; minimize or pseudonymize them and apply appropriate retention, access, and deletion controls
• Allowlist named fields — never serialize whole request/user/config objects; a JSON serializer will happily dump auth headers
• Redact at source, in the app — the pipeline is a second line of defense, not the first
• In regulated environments, every log line containing PII becomes a compliance obligation (retention, access control, right-to-erasure)

Where Instrumentation Lives (Architecture Placement)

In ports-and-adapters codebases, observability code has four homes — the four-tier model (full treatment: hexagonal-architecture skill, resources/cross-cutting-concerns.md):

1. Technical telemetry → adapters. Request/response logging, SQL timings, retries, auto-instrumentation. Never in domain code.
2. Domain-significant observations → an explicit driven port (Domain Probe) or domain events. When an intermediate business fact matters ("which pricing rule fired") or the observation is a requirement (support logging, business metrics), the observability backend is a driven actor behind a per-capability, severity-free, fire-and-forget port — never a generic Logger port. Where domain events already exist, an observability subscriber beats a second channel.
3. Correlation and wide-event assembly → middleware/adapters only. The domain never sees a trace ID. Domain dimensions reach the wide event via result types, the probe, or events.
4. Instrumentation is tested behavior. A probe is a driven port; every driven port gets a fake (see Testing Observability below).

Even without hexagonal architecture, the same instinct applies: business logic returns data and announces facts; edges translate those into telemetry.

Testing Observability

Instrumentation is behavior, so it is test-driven like behavior (see the tdd and testing skills).

• In-memory exporters — the OTel SDKs ship them — let Vitest assert on every span, attribute, and status without network calls: "this request emits one canonical event containing pledge.rejection_reason", "this failure sets span status to error". See resources/testing-telemetry.md.
• Telemetry ports get fakes, same as any driven port — a recording fake accumulates observations and tests assert on them through the public API. Worked example: hexagonal-architecture skill, resources/testing-hex-arch.md.
• Alert rules are code. Where the stack allows (Prometheus rule unit tests, SLO-as-code tools), test that the burn-rate expression fires on synthetic data.
• Mutation-testing note: an unasserted probe/telemetry call is a surviving-mutant farm — which is itself the argument for asserting observations.

Honest limits: you cannot meaningfully unit-test sampling percentages, Collector pipelines, or backend retention. Verify those in a staging environment with a real Collector — and note that sampling config is itself a parity surface (staging at 100%, prod at 1% behave differently under debugging).

Frontend Note (Out of Scope for v1)

Browser observability is RUM (real-user monitoring) with Core Web Vitals as the standard SLIs — LCP ≤ 2.5s, INP ≤ 200ms, CLS ≤ 0.1 at the 75th percentile of real page loads; lab tools are "not a substitute for field measurement" (web.dev). This skill covers services; for the user-facing half see the core-web-vitals and performance skills where installed.

Anti-Patterns

Boundaries

Verification Checklist

• Every request emits exactly one canonical wide event, including on the exception path
• Privacy-approved high-cardinality identifiers (subject, request, build, tenant) live on events/spans, never on metric labels
• Metric labels use bounded normalized allowlists; unknown request values map to _OTHER or are omitted, and raw attacker-controlled values never become labels
• service.name and resource attributes are set; semantic-convention names used where they exist
• Trace context (W3C traceparent) propagates across every service hop and into every log record
• OTel SDK initializes before application code loads (--import), auto-instrumentation enabled
• Production telemetry uses the recommended Collector path, or a documented direct-export exception meets buffering, retry, backpressure, filtering, and operational requirements
• Sampling and Collector capacity meet a stated error-trace retention objective and loss budget; use unsampled collection when the objective cannot tolerate sampling loss
• Each user journey has a handful of ratio-based SLOs at most; latency thresholds and diagnostic percentiles are explicit, with an error budget policy
• Paging alerts are symptom-based and burn-rate-driven (multiwindow); each links a runbook
• No page fires for a condition with no immediate human action
• Telemetry contains no secrets; personal identifiers are necessary, policy-approved, minimized or pseudonymized, and fields are allowlisted and redacted at source
• Instrumentation is covered by tests (in-memory exporter or fake-probe assertions)
• Adding a metric label or new signal triggers a written cardinality/cost estimate