Observability
The test of instrumentation is a question: when this misbehaves at 3 a.m.,
can the person on call find out what and why without adding new code? Build
toward answering that, not toward volume.
Method
- Log events, with context, structured. Each log line is a fact with
the fields needed to act on it: timestamp, level, event name, the ids
that connect it to a request, user, or job, and the specifics ("import
failed", file, page, cause). Structured key-value beats prose because
the reader at 3 a.m. is a query, not a person scrolling.
- Choose levels by the reader's need: ERROR means someone should act;
WARN means worth noticing on a bad day; INFO narrates state changes
worth a timeline (started, migrated, deployed); DEBUG is for
development and off by default. An ERROR nobody acts on gets demoted,
because alarm fatigue is how real errors get missed.
- Propagate one correlation id from the edge through every hop, into
every log line and outbound call. The single most valuable observability
feature is being able to pull one request's whole story with one query.
- Measure the four that matter per service: traffic, error rate,
latency as percentiles (p50, p95, p99, because averages hide the pain),
and saturation (queue depth, pool usage, memory). Add per-feature
counters only where a business question needs them.
- Alert on symptoms, with a runbook. Page on what users feel (error
rate, latency budget burn), not on causes like CPU, which belong on
dashboards. Every alert states what to check first and what usually
fixes it; an alert without a next action is noise with a pager.
- Never log secrets or personal data. Tokens, passwords, keys, and
raw personal content stay out of logs structurally (redaction at the
logger), not by hoping call sites remember.
Litmus tests
- Given one failing request id, can you reconstruct its path and failure
cause from signals alone?
- Does every page in the last month correspond to something a human did?
- Can you tell the difference between "slow for everyone" and "slow for
one huge tenant" from the dashboards?
Boundaries
Instrumentation follows the codebase's existing framework and conventions;
a second logging system is a bug, not an improvement. Cost is real: sample
high-volume traces, cap cardinality on metric labels, and expire what
nobody queries.
1---2name: observability3description: Instrument software so production questions get answered from signals, not guesses. Use when adding logging, metrics, tracing, or alerts, or when a system is hard to debug in production.4---56# Observability78The test of instrumentation is a question: when this misbehaves at 3 a.m.,9can the person on call find out what and why without adding new code? Build10toward answering that, not toward volume.1112## Method13141. **Log events, with context, structured.** Each log line is a fact with15 the fields needed to act on it: timestamp, level, event name, the ids16 that connect it to a request, user, or job, and the specifics ("import17 failed", file, page, cause). Structured key-value beats prose because18 the reader at 3 a.m. is a query, not a person scrolling.192. **Choose levels by the reader's need:** ERROR means someone should act;20 WARN means worth noticing on a bad day; INFO narrates state changes21 worth a timeline (started, migrated, deployed); DEBUG is for22 development and off by default. An ERROR nobody acts on gets demoted,23 because alarm fatigue is how real errors get missed.243. **Propagate one correlation id** from the edge through every hop, into25 every log line and outbound call. The single most valuable observability26 feature is being able to pull one request's whole story with one query.274. **Measure the four that matter per service:** traffic, error rate,28 latency as percentiles (p50, p95, p99, because averages hide the pain),29 and saturation (queue depth, pool usage, memory). Add per-feature30 counters only where a business question needs them.315. **Alert on symptoms, with a runbook.** Page on what users feel (error32 rate, latency budget burn), not on causes like CPU, which belong on33 dashboards. Every alert states what to check first and what usually34 fixes it; an alert without a next action is noise with a pager.356. **Never log secrets or personal data.** Tokens, passwords, keys, and36 raw personal content stay out of logs structurally (redaction at the37 logger), not by hoping call sites remember.3839## Litmus tests4041- Given one failing request id, can you reconstruct its path and failure42 cause from signals alone?43- Does every page in the last month correspond to something a human did?44- Can you tell the difference between "slow for everyone" and "slow for45 one huge tenant" from the dashboards?4647## Boundaries4849Instrumentation follows the codebase's existing framework and conventions;50a second logging system is a bug, not an improvement. Cost is real: sample51high-volume traces, cap cardinality on metric labels, and expire what52nobody queries.
Amey-Thakur