Observability

Source: addyosmani/agent-skills - skills/observability-and-instrumentation, adapted.

Observability

Instrumentation is written alongside the feature, the same way tests are. A feature that ships without telemetry turns its first bug into archaeology instead of a query.

Not for diagnosing a failure happening right now: that is /debug. This skill is what makes /debug fast next time.

Process

why-no-hook: skill workflow guidance; each step requires understanding the surrounding context (repo, task shape, prior state).

1. Write the on-call questions first

Telemetry without a question is noise. Before adding any instrumentation, write down 2-4 questions an on-call engineer will ask about this feature.

FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why? (provider error? timeout? validation?)
3. Is the payment provider slower than usual?

Every signal below must help answer one of these questions. If you cannot name the questions, you are not ready to instrument: you will log everything and learn nothing.

2. Pick the signal per question

Metrics tell you that something is wrong. Traces tell you where. Logs tell you why.

3. Structured logging

Log events, not prose. Every log line is a JSON object with a stable event name and machine-readable fields.

/* BAD: string interpolation, unqueryable */
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

/* GOOD: stable event name plus structured fields */
logger.warn({ event: 'payment_failed', paymentId: id, provider: 'stripe', errorCode: err.code, attempt: n }, 'payment failed');

Levels: error means an invariant broke and someone may need to act. warn means degraded but handled (retry succeeded, fallback used). info marks a significant business event. debug stays off in production.

Correlation IDs are mandatory. Generate or accept a request ID at the system boundary. Attach it to every log line, span, and outbound call.

app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
  req.log = logger.child({ requestId: req.id });
  res.setHeader('x-request-id', req.id);
  next();
});

Never log secrets, tokens, passwords, or full PII. Allowlist fields; never log whole request bodies (review-time: leak detection needs the field semantics, not a pattern)

4. Metrics

Instrument RED on every endpoint and every external dependency: Rate, Errors, Duration as a latency histogram. For resources (queues, pools, hosts) use USE: Utilization, Saturation, Errors.

Cardinality is the failure mode. Every unique label combination is a separate time series, so labels come from small fixed sets.

OK as label:    route="/api/tasks/:id"   status_class="5xx"   provider="stripe"
NEVER a label:  user_id, email, request_id, full URL, error message text

Averages never, percentiles always: an average hides the 1% of users having a terrible time. Use histograms and read p50/p95/p99.

5. Tracing

Use OpenTelemetry. Auto-instrumentation covers HTTP, gRPC, and common DB clients with near-zero code.

/* tracing.ts, imported before anything else */
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({ serviceName: 'checkout-service', instrumentations: [getNodeAutoInstrumentations()] });
sdk.start();

Add manual spans only around meaningful internal units of work, with the attributes on-call will filter by. Propagate context across every async boundary (HTTP headers, queue message metadata), or the trace dies at the gap. Sample head-based at a low rate; keep 100% of errors when the backend supports tail sampling.

6. Alerting

Alert on symptoms users feel, never on causes. Cause alerts (CPU at 85%, one pod restarted) fire when nothing is wrong and miss the failures you did not predict.

Rules for every alert you create:

1. Actionable: if the response is "ignore it, it self-heals", delete the alert (review-time: see section note)
2. Links to a runbook, even three lines: what it means, first query, escalation path (review-time: see section note)
3. Threshold and duration justified by the SLO or historical data, never a guess (review-time: see section note)
4. Two severities only: page (users hurt, act now) and ticket (act this week) (review-time: see section note)

7. Verify the telemetry itself

Instrumentation is code; it can be wrong. Trigger the paths and look at the actual output:

• Force an error in staging, then find it in the logs by requestId with structured fields.
• Send test traffic; confirm metric series appear with the expected labels and sane values.
• Follow one request across services in the tracing UI: no broken spans.
• Fire each new alert once with a temporarily lowered threshold; confirm the channel and the runbook link.