Logging Best Practices

This skill is adpated from "Logging sucks. And here's how to make it better." by Boris Tane.

When helping with logging, observability, or debugging strategies, follow these principles:

Core Philosophy

• Logs are optimized for querying, not writing — always design with debugging in mind
• Context is everything — a log without correlation IDs is useless in distributed systems
• Logs are for humans during incidents, not just for compliance or "just in case"
• If you can't filter and search your logs effectively, they provide zero value

Structured Logging Requirements

• Always use key-value pairs (JSON) instead of string interpolation
• Bad: "Payment failed for user 123"
• Good: {"event": "payment_failed", "user_id": "123", "reason": "insufficient_funds", "amount": 99.99}
• Structured logs are machine-parseable, enabling aggregation, alerting, and dashboards

Required Fields for Every Log Event

• timestamp — ISO 8601 with timezone (e.g., 2025-01-24T20:00:00Z)
• level — debug, info, warn, error (be consistent, don't invent new levels)
• event — machine-readable event name, snake_case (e.g., user_login_success)
• request_id or trace_id — for correlating logs across a single request
• service — which service/application emitted this log
• environment — prod, staging, dev

Examples of High-Cardinality Fields (always include when available):

• user_id, org_id, account_id — who is affected
• request_id, trace_id, span_id — for distributed tracing
• order_id, transaction_id, job_id — domain-specific identifiers

These fields are what make logs actually queryable during incidents. Without them, you're grepping through millions of lines blindly.

Look for opportunities for high-cardinality fields that can help you identify the root cause of an issue quickly.

Context Propagation

• Pass trace/request IDs through all service boundaries (HTTP headers, message queues, etc.)
• Downstream services must inherit correlation IDs from upstream
• Use middleware or interceptors to automatically inject context into every log
• For async jobs, store and restore the original request context

Log Levels — Use Them Correctly:

• debug — Verbose details for local development, usually disabled in production
• info — Normal operations worth recording (user actions, job completions, deploys)
• warn — Something unexpected happened but the system handled it (retries, fallbacks)
• error — Something failed and likely needs human attention (exceptions, failed requests)

Don't log errors for expected conditions (e.g., user enters wrong password)

What to Log:

• Request entry and exit points (with duration)
• State transitions (order created → paid → shipped)
• External service calls (with latency and response codes)
• Authentication and authorization events
• Background job starts, completions, and failures
• Retry attempts and circuit breaker state changes

What NOT to Log:

• Sensitive data (passwords, tokens, PII, credit card numbers)
• Logs inside tight loops (will generate millions of useless entries)
• Success cases that provide no debugging value
• Redundant information already captured by infrastructure (load balancer logs, etc.)

Naming Conventions:

• Be consistent across all services — agree on field names as a team
• Use snake_case for field names: user_id, not userId or user-id
• Use past-tense verbs for events: payment_completed, not complete_payment
• Prefix events by domain when helpful: auth.login_failed, billing.invoice_created

Performance Considerations:

• Use sampling for high-volume debug logs in production
• Avoid logging inside hot paths unless absolutely necessary
• Buffer and batch log writes to reduce I/O overhead
• Consider log levels that can be changed at runtime without redeploying

During Incidents:

• Your logs should answer: Who was affected? What failed? When? Why?
• If you can't answer these within 5 minutes of querying, your logging strategy needs work
• Post-incident: add the logs you wished you had