Observability When To Add

Observability & Instrumentation — Directed, Not Preventive

Industry principle (Google SRE Book, Honeycomb, Charity Majors): instrumentation carries maintenance cost proportional to its value. Every metric is code to maintain, schema that can break, storage cost, and signal to interpret. You instrument when you have confirmed pain, not "just in case."

When to add observability

• Performance bug you cannot locate (classic in multi-threaded, async, or distributed systems)
• External SLA or performance requirement demands metrics
• System is in production with real user traffic
• Flow analysis is part of the product itself (user journey tracking, etc.)

When NOT to add observability

• "Good practice" or preventive coverage before pain is identified
• Small side projects in active development
• Before you have identified a concrete bottleneck or failure mode
• To satisfy an abstract rule like "instrument everything"

What to use when you decide to instrument

• Production / real traffic → OpenTelemetry (CNCF graduated project) + collector + backend (Honeycomb, Tempo, Jaeger, or cloud equivalent)
• Local development / single-user → structured JSON logging + jq/grep. Add correlation_id if the call crosses process boundaries.
• Spot performance hunt → use language builtins (console.time/console.timeEnd in JavaScript, time.perf_counter() in Python, etc.). Discard after resolving the issue.

Universal principles when instrumenting

Instrument boundaries, not internals

• Entry points (API handlers, RPC endpoints)
• Network calls (HTTP, gRPC, database queries)
• External service boundaries
• NOT every function, NOT every variable assignment

Aggressive sampling in production

• Default: 1-10% of requests
• 100% only in development or for high-value (error) events
• Sampling reduces noise and cost without losing signal on aggregate behavior

Telemetry failure must not cascade

• All log writes go in try/catch blocks (silent failure okay)
• If the observability system is down, the application runs normally
• Observability is about understanding running systems, not blocking them

Never log PII, tokens, or raw payloads

• Log summaries instead of bodies (e.g., "request 256 bytes to /api/v1/users" not the JSON)
• Redact or hash sensitive fields
• Assume logs are readable by ops teams and retained for weeks

Use verb:noun naming for events

• Examples: fetch:external_api, query:database, publish:event
• This naming schema makes grep work reliably months later
• Easier to discover related events and trace flows

Use monotonic clocks for duration

• performance.now() (JavaScript), time.perf_counter() (Python), System.nanoTime() (Java)
• NOT wall-clock time (Date.now(), time.time()) — wall clocks can jump backward
• Duration must be reliable across system-clock corrections

Red flags — when you're over-instrumenting

• Observability code is more complex than the business logic it instruments
• You added metrics months ago and have never queried them
• The instrumentation is "just in case" with no corresponding monitoring/alerting
• Log volume exceeds bandwidth or storage budget
• You notice yourself filtering/ignoring metrics because they're too noisy

Canonical references

• Google SRE Book, chapters 6 (Monitoring Distributed Systems) and 12 (Effective Troubleshooting)
• Charity Majors, "Observability is for Unknown Unknowns"
• OpenTelemetry specification: https://opentelemetry.io/
• Honeycomb Observability Guide: https://www.honeycomb.io/