Observability Instrumentation

When to Use

• Use when: production issues are hard to diagnose because logs are unstructured
• Use when: a request spans multiple services and you can't follow the path
• Use when: adding metrics to a new service before its first deploy
• Use when: setting up OpenTelemetry for a project
• Do NOT use for: frontend performance monitoring — use web-performance

The Three Pillars

All three are needed. Metrics tell you something is wrong; logs tell you what; traces tell you where.

Structured Logging

Every log line must be machine-parseable JSON. No free-text concatenation.

Required fields on every log line

{
  "timestamp": "2025-05-21T10:00:00.123Z",
  "level":     "INFO",
  "service":   "order-service",
  "trace_id":  "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id":   "00f067aa0ba902b7",
  "message":   "Order placed successfully",
  "order_id":  "ord-123",
  "user_id":   "usr-456",
  "duration_ms": 42
}

Log levels — use consistently

Rules:

• Never log PII (email, name, card number, tokens) — log IDs only
• Never interpolate secrets into log messages
• Include trace_id on every line — enables cross-service correlation

Distributed Tracing

Concepts

Trace:  one request's full journey across all services
Span:   one operation within a trace (service call, DB query, external API)
Parent span: the span that initiated a child span

Trace:   [order-service: handle POST /orders] (root span)
  └── Span: [payment-service: charge card]
  └── Span: [inventory-service: reserve stock]
       └── Span: [DB: UPDATE inventory]

Propagation — pass trace context in HTTP headers

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             version-traceId-spanId-flags

Use W3C TraceContext standard. Always forward traceparent on all outbound calls.

OpenTelemetry setup (minimal)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

provider = TracerProvider()
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("order-service")

with tracer.start_as_current_span("place-order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("user.id", user_id)
    # ... do work

Metrics Naming Conventions

Follow Prometheus naming conventions — other systems largely adopt the same.

{namespace}_{subsystem}_{name}_{unit}

Examples:
  http_requests_total              (counter — always _total suffix)
  http_request_duration_seconds    (histogram — use base units: seconds not ms)
  db_connections_active            (gauge — current state)
  order_processing_errors_total    (counter)
  cache_hit_ratio                  (gauge — 0.0 to 1.0)

Metric types

Mandatory labels (cardinality warning)

http_requests_total{method="POST", status="200", endpoint="/orders"}

Never use user IDs, order IDs, or any high-cardinality value as a label — creates metric explosion.

Instrumentation Checklist

Every production service must have:

□ Structured JSON logs with timestamp, level, service, trace_id on every line
□ trace_id injected into all log lines (MDC / context propagation)
□ Outbound HTTP calls: forward traceparent header
□ Inbound requests: extract trace context or start new root span
□ Metrics: request count, error count, duration histogram per endpoint
□ Alerts wired to error rate and latency metrics (see slo-design)
□ No PII in logs — IDs only

Anti-Fake-Pass Rules

Before claiming a service is observable, you MUST show:

• Logs are structured JSON — no free-text concatenation
• trace_id present on every log line and propagated to downstream calls
• At least 3 metrics: request count, error count, latency histogram
• OpenTelemetry (or equivalent) trace spans on outbound calls and DB queries
• PII verified absent from log fields (spot-check 5 representative log lines)

Reference: gates/anti-fake-pass-gate.md