martinholovsky SOTA Observability & Reliability
Purpose
Make every production system answerable. Two questions define success:
- "Why is this request slow/failing?" — answerable for any single request from a trace ID, without adding new instrumentation.
- "What broke at 3am?" — answerable from symptom-based alerts that page only when users are hurt, each linked to a runbook and a dashboard that narrows cause in minutes.
This skill covers structured logging, metrics, distributed tracing, SLOs and alerting, and operational readiness — both how to build them correctly and how to audit them adversarially. Telemetry is a product with users (on-call engineers) and costs (storage, cardinality, attention). Treat both.
BUILD mode
When writing or modifying code, apply the rules files as design constraints, not afterthoughts. Workflow:
- Identify the signal need before coding. For each new endpoint, job, or consumer: which SLI does it affect, what one wide event describes a unit of work, what spans bound its external calls.
- Instrument with OpenTelemetry API (not vendor SDKs) in libraries; configure SDK/exporters only at the application entry point. Follow OTel semantic conventions for names and attributes.
- Emit one canonical wide event per request/job at completion, carrying trace_id, outcome, durations, and business context. Debug logs are supplementary, sampled, and disposable.
- Propagate context everywhere: W3C
traceparentover HTTP, injected into queue message headers, restored in consumers and scheduled jobs. - Redact at the logger, never at call sites. Denylist+allowlist serializers for PII/secrets; fail closed on unknown object dumps.
- Budget cardinality. Every metric label must have a known, bounded value set. No IDs, no URLs, no user input in labels.
- Ship the operational surface with the feature: health endpoints with correct liveness/readiness semantics, dashboard panels answering the questions the feature raises, burn-rate alerts wired to the SLO, runbook entry for each new alert.
- Verify by simulation: kill a dependency, send a slow request, trigger an error — confirm the trace, the wide event, the metric, and the alert all show it, and that they cross-link (exemplars, trace_id in logs).
AUDIT mode
Assess an existing codebase/deployment. Read rules/06 first for the full
playbook; sample real code paths, do not trust README claims.
Severity conventions:
| Severity | Meaning | Examples |
|---|---|---|
| CRITICAL | Blind during incidents, or telemetry is itself a hazard | Secrets/PII in logs; no error visibility at all; liveness check hits the database (restart storms); unauthenticated debug/pprof endpoints |
| HIGH | Materially slows MTTR or breaks at scale | No correlation/trace IDs; unbounded label cardinality; cause-based paging alerts with no runbooks; readiness == liveness; percentiles averaged across instances |
| MEDIUM | Degrades signal quality or cost discipline | Wrong log levels (ERROR for expected events); no exemplars; head-only sampling losing all error traces; dashboards as vanity walls; no log sampling on hot paths |
| LOW | Hygiene and polish | Inconsistent field names; missing OTel semantic conventions; unpinned dashboard queries; noisy Sentry grouping |
Finding format (one per finding):
[SEVERITY] <short title>
Where: <file:line, config path, or dashboard/alert name>
Evidence: <exact code/config snippet or observed behavior>
Impact: <what fails during an incident or at scale, concretely>
Fix: <specific change, with code/config if short>
Effort: <S/M/L>
Conclude every audit with the two-question verdict: can on-call currently answer "why is this request slow?" and "what broke at 3am?" — YES/PARTIAL/NO, with the shortest path to YES.
Rules index
| File | Read this when... |
|---|---|
rules/01-structured-logging.md |
Writing or reviewing log statements, choosing levels, designing wide events/canonical log lines, configuring redaction, sampling, or controlling log spend |
rules/02-metrics.md |
Adding Prometheus/OTel metrics, choosing counter vs gauge vs histogram, designing labels, computing percentiles, applying RED/USE, linking metrics to traces via exemplars |
rules/03-tracing.md |
Instrumenting with OpenTelemetry, deciding what gets a span, propagating context across HTTP/queues/jobs, choosing head vs tail sampling, using (or avoiding) baggage |
rules/04-slos-alerting.md |
Defining SLIs/SLOs, error budgets, writing burn-rate alerts, reviewing alert quality, fighting alert fatigue, deciding page vs ticket |
rules/05-operational-readiness.md |
Implementing health endpoints, exposing graceful degradation, securing debug endpoints, continuous profiling, Sentry-style error tracking, building dashboards, edge access logs as a decision-grade signal, keeping test-written telemetry out of the sink production writes — and the question with no instrument, where a proxy measurement silently answers a different one |
rules/06-audit-playbook.md |
Auditing a codebase's observability posture end-to-end; common gaps catalog; scoring and reporting |
Top 10 non-negotiables
- Every log line carries a trace/correlation ID. A log you cannot join to a request is gossip, not evidence.
- ERROR means a human must act. If nobody should be woken or ticketed, it is WARN or below. Level discipline is alert discipline upstream.
- No secrets or PII in telemetry — enforced at the logger/exporter, not by call-site vigilance. Redaction is infrastructure, not convention.
- One wide event per unit of work (request/job/message) with outcome, duration, and business context — the canonical log line you grep at 3am.
- Metric labels are bounded. No user IDs, emails, raw URLs, or free text. Cardinality explosions take down the monitoring you need most.
- Never average percentiles. Aggregate histograms, then compute quantiles. A dashboard of avg(p99) is fiction.
- OpenTelemetry API in libraries, SDK only at the edge. W3C
traceparentpropagated across every HTTP hop, queue, and async job. - Liveness checks process health only; readiness checks dependencies. Conflating them turns one slow dependency into a cluster-wide restart storm.
- Every page is actionable, symptom-based, and runbook-linked. Alert on user pain (SLO burn rate, multi-window), not on causes (CPU, pod restarts).
- Telemetry has a budget. Sample debug logs and traces deliberately (tail-sample to keep errors/slow), review cost monthly, delete signals nobody queries.