Observability and Instrumentation
Start from the question an operator must answer, then emit the smallest safe
signal that answers it.
Design sequence
- Name the user or business signal, SLO/error budget, failure mode, and
responder.
- Map the request or job across boundaries and choose correlation/trace IDs,
cardinality, sampling, and clock conventions.
- Define structured logs for state transitions and failures, metrics for
rates/latency/saturation, and traces for cross-service causality.
- Include useful dimensions without high-cardinality or secret data. Redact
tokens, PII, payloads, and credentials by default.
- Add alerts with actionable thresholds, runbook links, ownership, deduping,
and a recovery path. Avoid alerts that merely report expected retries.
- Verify the signal in a safe environment and test the failure path that
should emit it.
Prepare operational baselines
For release and incident use, record the pre-change baseline, freshness,
expected variance, promotion or paging threshold, and recovery signal. Include
one user or business outcome alongside technical health where possible. A
dashboard without a decision owner and action threshold is reference material,
not a guardrail.
Preserve a correlation path from release identifier to request, job, dependency,
and user-visible outcome. Ensure incident responders can distinguish absent
traffic, stale telemetry, collector failure, and healthy zero values.
Read signal-design.md. Instrumentation must not
change correctness or become the only source of truth for business data.
Completion condition
An operator can detect the important failure, correlate it across boundaries,
understand the likely cause, and follow a bounded response without exposing
sensitive data. release-engineering can consume the baseline as a promotion
gate; incident-response can consume it as live evidence.
1---2name: observability-and-instrumentation3description: Designs or reviews logs, metrics, traces, alerts, correlation, dashboards, and diagnostic instrumentation so failures and performance changes can be detected and explained at component boundaries. Use for observability work, incident readiness, instrumentation, or production diagnostics. Not for fixing a specific incident before reproducing it or for adding noisy logging without an operational question.4---56# Observability and Instrumentation78Start from the question an operator must answer, then emit the smallest safe9signal that answers it.1011## Design sequence12131. Name the user or business signal, SLO/error budget, failure mode, and14 responder.152. Map the request or job across boundaries and choose correlation/trace IDs,16 cardinality, sampling, and clock conventions.173. Define structured logs for state transitions and failures, metrics for18 rates/latency/saturation, and traces for cross-service causality.194. Include useful dimensions without high-cardinality or secret data. Redact20 tokens, PII, payloads, and credentials by default.215. Add alerts with actionable thresholds, runbook links, ownership, deduping,22 and a recovery path. Avoid alerts that merely report expected retries.236. Verify the signal in a safe environment and test the failure path that24 should emit it.2526## Prepare operational baselines2728For release and incident use, record the pre-change baseline, freshness,29expected variance, promotion or paging threshold, and recovery signal. Include30one user or business outcome alongside technical health where possible. A31dashboard without a decision owner and action threshold is reference material,32not a guardrail.3334Preserve a correlation path from release identifier to request, job, dependency,35and user-visible outcome. Ensure incident responders can distinguish absent36traffic, stale telemetry, collector failure, and healthy zero values.3738Read [signal-design.md](references/signal-design.md). Instrumentation must not39change correctness or become the only source of truth for business data.4041## Completion condition4243An operator can detect the important failure, correlate it across boundaries,44understand the likely cause, and follow a bounded response without exposing45sensitive data. `release-engineering` can consume the baseline as a promotion46gate; `incident-response` can consume it as live evidence.
thiientv