Instrument Backend Observability

Instrument Backend Observability

Purpose

Make backend services diagnosable in production by standardizing logs, error tracking, metrics, and tracing.

When to use

Use this skill when you are:

• Adding new endpoints or background jobs that require monitoring
• Debugging production incidents (5xx spikes, latency regressions)
• Integrating an error tracker or APM solution
• Standardizing log formats and correlation IDs

Inputs

• The runtime environment(s) and deployment model
• Current logging and monitoring stack (if any)
• What “good” looks like: SLOs, latency targets, error budgets

Outputs

• A consistent logging and error tracking plan
• Standard fields for correlation and debugging
• A minimal alert strategy for critical signals

Core rules

• Unknown errors MUST be captured by an error tracker (or equivalent) with context.
• Logs MUST be structured and SHOULD include a correlation/request ID.
• Sensitive data MUST NOT be logged (tokens, passwords, secrets, raw PII beyond what is required).
• Observability MUST NOT change business behavior (instrumentation should be side-effect free).

Recommended signals

• Errors
  • rate of 5xx
  • rate of domain-specific 4xx (for detecting client issues or abuse)
• Latency
  • p50/p95/p99 per endpoint
• Saturation
  • CPU, memory, DB connection pool utilization
• Traffic
  • request volume per endpoint

Steps

1. Ensure a request/correlation ID exists for every request.
2. Add structured logs at key boundaries:
   • request start/end (method, path, status, duration)
   • key domain actions (entity IDs, operation names)
3. Capture exceptions with context:
   • endpoint name
   • user/tenant identifiers (redacted as needed)
   • correlation ID
4. Add metrics for:
   • request duration
   • error counts
5. Define alerts for:
   • sustained 5xx rate
   • sustained latency regression
6. Verify by simulating:
   • a known operational error
   • an unknown exception

Verification

• All requests have a correlation/request ID in logs
• Structured logs include method, path, status, and duration
• Exceptions are captured with correlation ID and endpoint context
• Sensitive data (tokens, passwords, PII) is not present in logs
• Alerts fire for sustained 5xx rates (test with simulated errors)
• Latency metrics are recorded per endpoint

Boundaries

• MUST NOT log secrets, tokens, passwords, or raw PII
• MUST NOT allow observability code to change business behavior
• MUST NOT create high-cardinality metric labels (e.g., user IDs as labels)
• SHOULD NOT log request/response bodies in production (except for debugging)
• SHOULD NOT rely solely on logs for error tracking (use a dedicated tracker)
• SHOULD NOT skip correlation ID propagation in async operations

Included assets

• Templates: ./templates/ includes recommended log fields and exception capture patterns.
• Examples: ./examples/ includes incident triage checklists.