Observability Instrumentation

Observability Instrumentation

Use when scale-readiness-review/resilience-patterns says "add a metric"; use debug-runtime to diagnose live failures.

Workflow

1. Pick the path (payment, payout, settlement, reconciliation, hot endpoint); list the questions ops will ask (did X succeed, failure rate/p95, queue depth, reconciliation drift).
2. Log: structured key-value/JSON, correlation id, stable business keys (transaction, account, idempotency key, provider ref, tenant) — never secrets/PANs/credentials; log decisions/transitions and call boundaries.
3. Metrics via the repo's client:
   • RED: Rate, Errors, Duration (histogram) for request-driven paths.
   • USE: Utilization, Saturation, Errors for resources (pools, workers, connections).
   • Domain signals: queue depth/age, settlement lag, retry counts, breaker state, reconciliation drift, idempotency-conflict rate.
   • Label by operation/outcome; bound cardinality.
4. Trace: propagate context through jobs/queues/calls; span each external call with duration/status.
5. Alert on symptoms users feel (error rate, latency, backlog, drift); SLO thresholds/burn rates; name cause and next step; delete unused alerts.
6. Verify: trigger the path, confirm logs/metrics/traces appear and alerts fire on failure.

Guardrails

• Do not log secrets, full card numbers, credentials, or unmasked PII.
• Do not use unbounded label cardinality (raw ids, emails).
• Do not rely on average latency; use histograms/percentiles.
• Do not add alerts without an owner and a response action.
• Do not over-instrument the hot path; sample where appropriate.