# Adding Observability

> Instruments a change or service with logs, metrics, traces, and symptom alerts. Use when adding logs, metrics, traces, or an alert, or when the user says add observability, add monitoring, instrument this, we have no visibility, or we cannot tell whether it is working. Skip rollout threshold signals, which rollout-controls owns, and reading existing signals to find a cause, which debugging owns.

- Skill: `augments-labs/adding-observability` (Agent Skill)
- Install (CLI): `npx get-skill add augments-labs/adding-observability`
- Raw SKILL.md: https://api.skillmd.com/api/skills/augments-labs/adding-observability/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: augments-labs (https://skillmd.com/u/augments-labs)
- Updated: 2026-11-11
- Page: https://skillmd.com/skills/augments-labs/adding-observability

---


# Adding Observability

This skill never adds a telemetry library or vendor on its own, never
renames a signal something already reads, and never decides who gets paged.

## When to use

- A change or a service needs signals so its operators can answer "is it
  working for users?" and "why not?".
- **Skip** the signals a staged rollout's thresholds read:
  `rollout-controls` owns them. A rollout that still cannot answer the two
  questions above comes here for the rest.
- **Skip** finding a cause from signals that already exist: `debugging` owns
  that, and the probes it adds are temporary.

## Step 1: Pick the questions, then the signals

1. Write the user-visible outcome the change exists to produce, and the two
   signals that answer the operator's questions: the success rate and the
   latency where the user feels them. A liveness check, CPU, memory, or
   "function called" answers neither question; add it only when something
   reads it.
2. For "why not?", name what an operator reads next when the success rate
   drops: the error kind by label, the log line with the request's context,
   and the trace across every hop the request takes.
3. Read the project's logging, metrics, and tracing facility, its naming
   convention, its unit convention, and the dashboard or query the operators
   already use. Emit through that, under those names. Add no library and no
   vendor.
4. If the project has no facility, the choice is the user's; ask and end the
   turn, and `clarifying-intent` owns what closes it:

   ```text
   {{project}} has no logging, metrics, or tracing facility. Where should the new signals go?

   1. Structured log lines on the process's standard output, which {{the host or platform}} already collects; counts and durations as fields of those lines
   2. A library or vendor you name: print which

   Recommendation: 1 — {{one sentence}}.
   ```

5. If a staged rollout is asked for, **REQUIRED SUB-SKILL:** invoke
   `rollout-controls` for the switch and the threshold signals, then return
   here for the signals that still lack an operator's question.

## Step 2: Build the signals

1. Give every metric a label set a human can list in advance: outcome, error
   kind, route template, operation. Never a user id, request id, raw URL
   path, email, or error message text.
2. Initialize every counter the alerts and ratios read to zero for each label
   set that can occur, at startup.
3. Record latency as a histogram whose buckets include every threshold
   someone will alert on. Never an average, and never a summary when
   instances are aggregated. Record the outcome and the duration on every
   way out of the operation: a raised error, a timeout, and a cancellation
   as well as a return.
4. Carry the trace or request id on every log line the request writes. Where
   the request crosses a queue, a thread pool, a scheduled job, or another
   process, inject the trace context into the message or carry it by hand.
5. If the facility samples traces at the start of the request, set it to keep
   errors when it allows that; if it does not, say in the report that error
   traces are sampled away.
6. Log an expected condition (a validation failure, a not-found, a retry
   that will succeed) below error level.
7. No secret, credential, token, request body, header, query string, or
   personal data in a signal's value, label, or log field.
8. In a process that exits before the facility collects (a batch job, a
   command-line run, a scheduled task), flush or push the signals before it
   exits.

## Step 3: Prove each signal is emitted

1. **REQUIRED SUB-SKILL:** invoke `test-driven-development` for each signal:
   a test that fails when the signal is missing, carries the wrong label, or
   carries a value from the forbidden list above.
2. For each signal the test cannot reach (a dashboard query, a trace across
   a hop, a sampled error), read it in a real environment and record where
   it was seen.

## Step 4: Alert, then hand off

1. Write each alert on a user-facing symptom (the success rate or latency
   from Step 1), never on a cause, with an owner and the first action the
   owner takes. Whether it pages anyone is the owner's decision; ask, never
   assume.
2. Fire each alert once on purpose (a test environment, a synthetic failure,
   or a lowered threshold restored afterwards) and record that it fired.
   Where the alert reaches a person or its rule lives in a shared
   environment, get the owner's go-ahead first or fire it through a route
   that pages no one.
3. If an existing metric name, label, or log field must change, every
   dashboard and alert that reads it is a caller. **REQUIRED SUB-SKILL:**
   invoke `deprecating-an-api` for the rename; never rename in place.
4. Report three lines and decide nothing:

   ```text
   Questions: working for users → {{signal}}; why not → {{signals}}
   Signals: {{n}} through {{facility}}, each seen in {{test | environment}}; unverified: {{list | none}}
   Alerts: {{n}}, each with owner and first action; fired on purpose: {{yes | which ones not, and who will}}
   ```

## Hard stops

- Adding a telemetry library or vendor the project does not already use
  without the user's answer to the Step 1 question.
- A secret, credential, token, request body, header, query string, or
  personal data in any signal's value, label, or log field.
- Renaming or removing a metric, label, or log field a dashboard or alert
  reads, outside `deprecating-an-api`.
- Calling a signal or an alert done when nobody has seen it emitted or fired.

## Gotchas

- A counter incremented only on failure has no series until the first
  failure, so a rate or a ratio over it returns no data rather than `0`, and
  an alert on "no data" never fires, or fires on every fresh instance —
  shows when an error-rate alert stays silent through an outage on a path
  that had never failed before.
- A summary's percentiles cannot be aggregated across instances, so a p99
  over a fleet is nonsense; a histogram's buckets are fixed at creation, a
  quantile between two boundaries is an interpolated guess, and changing
  the layout later breaks every query over the old series — shows when a
  300 ms alert on buckets of 250 ms and 500 ms fires on an estimate and
  nobody can say whether 300 ms was crossed.
- Unit conventions differ by facility and both are strict: one wants base
  units in a plural suffix (`_seconds`, `_bytes`, `_total` after the unit);
  another wants the unit in the metric's metadata (`s`, `By`) and not in the
  name. A duration in milliseconds under either breaks the fleet's shared
  queries — shows when `request_latency_ms` sits beside `_seconds` metrics
  and the dashboard's unit axis reads it a thousandfold wrong.

## Common mistakes

- Writing an alert for every metric: an alert without a first action is noise
  the owner silences along with the real ones.

