Adding Observability
This skill never adds a telemetry library or vendor on its own, never renames a signal something already reads, and never decides who gets paged.
When to use
- A change or a service needs signals so its operators can answer "is it working for users?" and "why not?".
- Skip the signals a staged rollout's thresholds read:
rollout-controlsowns them. A rollout that still cannot answer the two questions above comes here for the rest. - Skip finding a cause from signals that already exist:
debuggingowns that, and the probes it adds are temporary.
Step 1: Pick the questions, then the signals
Write the user-visible outcome the change exists to produce, and the two signals that answer the operator's questions: the success rate and the latency where the user feels them. A liveness check, CPU, memory, or "function called" answers neither question; add it only when something reads it.
For "why not?", name what an operator reads next when the success rate drops: the error kind by label, the log line with the request's context, and the trace across every hop the request takes.
Read the project's logging, metrics, and tracing facility, its naming convention, its unit convention, and the dashboard or query the operators already use. Emit through that, under those names. Add no library and no vendor.
If the project has no facility, the choice is the user's; ask and end the turn, and
clarifying-intentowns what closes it:{{project}} has no logging, metrics, or tracing facility. Where should the new signals go? 1. Structured log lines on the process's standard output, which {{the host or platform}} already collects; counts and durations as fields of those lines 2. A library or vendor you name: print which Recommendation: 1 — {{one sentence}}.If a staged rollout is asked for, REQUIRED SUB-SKILL: invoke
rollout-controlsfor the switch and the threshold signals, then return here for the signals that still lack an operator's question.
Step 2: Build the signals
- Give every metric a label set a human can list in advance: outcome, error kind, route template, operation. Never a user id, request id, raw URL path, email, or error message text.
- Initialize every counter the alerts and ratios read to zero for each label set that can occur, at startup.
- Record latency as a histogram whose buckets include every threshold someone will alert on. Never an average, and never a summary when instances are aggregated. Record the outcome and the duration on every way out of the operation: a raised error, a timeout, and a cancellation as well as a return.
- Carry the trace or request id on every log line the request writes. Where the request crosses a queue, a thread pool, a scheduled job, or another process, inject the trace context into the message or carry it by hand.
- If the facility samples traces at the start of the request, set it to keep errors when it allows that; if it does not, say in the report that error traces are sampled away.
- Log an expected condition (a validation failure, a not-found, a retry that will succeed) below error level.
- No secret, credential, token, request body, header, query string, or personal data in a signal's value, label, or log field.
- In a process that exits before the facility collects (a batch job, a command-line run, a scheduled task), flush or push the signals before it exits.
Step 3: Prove each signal is emitted
- REQUIRED SUB-SKILL: invoke
test-driven-developmentfor each signal: a test that fails when the signal is missing, carries the wrong label, or carries a value from the forbidden list above. - For each signal the test cannot reach (a dashboard query, a trace across a hop, a sampled error), read it in a real environment and record where it was seen.
Step 4: Alert, then hand off
Write each alert on a user-facing symptom (the success rate or latency from Step 1), never on a cause, with an owner and the first action the owner takes. Whether it pages anyone is the owner's decision; ask, never assume.
Fire each alert once on purpose (a test environment, a synthetic failure, or a lowered threshold restored afterwards) and record that it fired. Where the alert reaches a person or its rule lives in a shared environment, get the owner's go-ahead first or fire it through a route that pages no one.
If an existing metric name, label, or log field must change, every dashboard and alert that reads it is a caller. REQUIRED SUB-SKILL: invoke
deprecating-an-apifor the rename; never rename in place.Report three lines and decide nothing:
Questions: working for users → {{signal}}; why not → {{signals}} Signals: {{n}} through {{facility}}, each seen in {{test | environment}}; unverified: {{list | none}} Alerts: {{n}}, each with owner and first action; fired on purpose: {{yes | which ones not, and who will}}
Hard stops
- Adding a telemetry library or vendor the project does not already use without the user's answer to the Step 1 question.
- A secret, credential, token, request body, header, query string, or personal data in any signal's value, label, or log field.
- Renaming or removing a metric, label, or log field a dashboard or alert
reads, outside
deprecating-an-api. - Calling a signal or an alert done when nobody has seen it emitted or fired.
Gotchas
- A counter incremented only on failure has no series until the first
failure, so a rate or a ratio over it returns no data rather than
0, and an alert on "no data" never fires, or fires on every fresh instance — shows when an error-rate alert stays silent through an outage on a path that had never failed before. - A summary's percentiles cannot be aggregated across instances, so a p99 over a fleet is nonsense; a histogram's buckets are fixed at creation, a quantile between two boundaries is an interpolated guess, and changing the layout later breaks every query over the old series — shows when a 300 ms alert on buckets of 250 ms and 500 ms fires on an estimate and nobody can say whether 300 ms was crossed.
- Unit conventions differ by facility and both are strict: one wants base
units in a plural suffix (
_seconds,_bytes,_totalafter the unit); another wants the unit in the metric's metadata (s,By) and not in the name. A duration in milliseconds under either breaks the fleet's shared queries — shows whenrequest_latency_mssits beside_secondsmetrics and the dashboard's unit axis reads it a thousandfold wrong.
Common mistakes
- Writing an alert for every metric: an alert without a first action is noise the owner silences along with the real ones.