Observability

Observability

A service is not done until someone who has never read its code can answer "is it healthy right now" from its telemetry alone. That means structured logs that a platform can index and correlate, the four golden signals tracked per service, and an explicit SLO with an error budget, not "we'll notice if it breaks."

Phase 1: Structured logging

• Every log line is a structured record (JSON or the platform's structured format), not a hand-built string; fields are queryable and indexable; a concatenated string is only greppable.
• Every log line inside a request/job carries a correlation ID (trace ID / request ID) so a slow request, its trace, and its log lines can be joined without copy-pasting timestamps.
• Log levels mean something consistent across the codebase: error = needs human attention, warn = degraded but self-recovering, info = significant state change (request completed, job started), debug = development-time detail, off by default in production.
• Never log a secret, full payload, or PII field directly; log its presence or a redacted/hashed form (same redaction discipline as diagnosing-bugs and secure-coding).
• Log the outcome, not just the attempt: "started X" without a matching "X succeeded / X failed (reason)" leaves a gap no dashboard can fill in later.

Phase 2: The four golden signals

Every service exposes these four, regardless of what else it tracks:

For an API or service boundary specifically, the same four collapse into the RED method (Rate, Errors, Duration) as the minimum viable dashboard for that endpoint.

Phase 3: SLI, SLO, error budget

• SLI (Service Level Indicator): the specific, measurable signal: "proportion of requests completed in under 300ms", "proportion of requests returning non-5xx." Pick SLIs from Phase 2's signals; don't invent a new metric nobody dashboards.
• SLO (Service Level Objective): a target on that SLI over a window: "99.9% of requests succeed, measured over 30 days." The number is a product/business decision (how much unreliability users tolerate), not a technical maximum.
• Error budget: 1 − SLO over the window. It is spent by every failure, deploy-caused or not. A team with budget left ships faster; a team that has burned its budget freezes risky changes until it recovers. This is what turns "reliability" from a vague goal into a concrete, checkable number that also unblocks shipping speed when things are healthy.
• If a service cannot state its SLI and SLO in one sentence, it is not ready to call itself monitored; "we have dashboards" is not the same as "we know what healthy means."

Phase 4: Alerting and runbooks

• Alert on symptom, not cause: alert when the SLO is at risk (error budget burn rate, latency SLI breach) rather than on every underlying anomaly; cause-based alerts multiply noise and train people to ignore pages.
• Every alert that can page a human links a runbook: what this alert means, what to check first, what mitigates it, when to escalate. An alert with no runbook is a 3am guessing game.
• Tune alert thresholds against the error budget's burn rate, not a fixed absolute number picked once and forgotten; a slow burn over days and a fast burn over minutes are different incidents needing different urgency.

Done when

• Logs are structured, correlated by request/trace ID, and carry no secret or raw PII.
• The service exposes latency (success/failure split), errors, traffic, and saturation.
• The service has a stated SLI and SLO, and the error budget is a number someone could quote.
• Every human-paging alert links a runbook, and thresholds are tied to error-budget burn rate, not an arbitrary constant.
• Generated prose uses commas, colons, periods, or reworded sentences instead of em dashes.