# Observability Instrumentation

> Adds structured logging, metrics, tracing, and actionable alerts so financial systems are explainable in production. Use for hard-to-diagnose incidents, new money-moving paths, or drift/queue-depth/latency/failure-rate observability.

- Skill: `rockclaver/observability-instrumentation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add rockclaver/observability-instrumentation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rockclaver/observability-instrumentation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: rockclaver (https://skillmd.com/u/rockclaver)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rockclaver/observability-instrumentation

---


# Observability Instrumentation

Use when `scale-readiness-review`/`resilience-patterns` says "add a metric"; use `debug-runtime` to diagnose live failures.

## Workflow

1. **Pick the path** (payment, payout, settlement, reconciliation, hot endpoint); list the questions ops will ask (did X succeed, failure rate/p95, queue depth, reconciliation drift).
2. **Log**: structured key-value/JSON, correlation id, stable business keys (transaction, account, idempotency key, provider ref, tenant) — never secrets/PANs/credentials; log decisions/transitions and call boundaries.
3. **Metrics** via the repo's client:
   - **RED**: Rate, Errors, Duration (histogram) for request-driven paths.
   - **USE**: Utilization, Saturation, Errors for resources (pools, workers, connections).
   - **Domain signals**: queue depth/age, settlement lag, retry counts, breaker state, reconciliation drift, idempotency-conflict rate.
   - Label by operation/outcome; bound cardinality.
4. **Trace**: propagate context through jobs/queues/calls; span each external call with duration/status.
5. **Alert** on symptoms users feel (error rate, latency, backlog, drift); SLO thresholds/burn rates; name cause and next step; delete unused alerts.
6. **Verify**: trigger the path, confirm logs/metrics/traces appear and alerts fire on failure.

## Guardrails

- Do not log secrets, full card numbers, credentials, or unmasked PII.
- Do not use unbounded label cardinality (raw ids, emails).
- Do not rely on average latency; use histograms/percentiles.
- Do not add alerts without an owner and a response action.
- Do not over-instrument the hot path; sample where appropriate.