Structured Logging
Core Philosophy
- Logs are optimized for querying, not writing. Design with debugging in mind.
- A log without correlation IDs is useless in distributed systems
- If you can't answer "Who was affected? What failed? When? Why?" within 5 minutes, logging needs work
Structured Format
Always use key-value pairs (JSON), never string interpolation.
{
"event": "payment_failed",
"user_id": "123",
"reason": "insufficient_funds",
"amount": 99.99,
"timestamp": "2025-01-24T20:00:00Z",
"level": "error",
"service": "billing",
"request_id": "req_abc123"
}
Required Fields
Every log event MUST include:
| Field |
Format |
Example |
timestamp |
ISO 8601 with timezone |
2025-01-24T20:00:00Z |
level |
debug, info, warn, error |
info |
event |
snake_case, past tense |
user_login_succeeded |
request_id or trace_id |
UUID or prefixed ID |
req_abc123 |
service |
Service/app name |
api-gateway |
environment |
prod, staging, dev |
prod |
High-Cardinality Fields
Include these when available. They make logs queryable during incidents:
| Category |
Fields |
| Identity |
user_id, org_id, account_id |
| Tracing |
request_id, trace_id, span_id |
| Domain |
order_id, transaction_id, job_id |
Rule: Look for domain-specific identifiers that help isolate issues to specific entities.
Log Levels
| Level |
When to Use |
Example |
debug |
Verbose local dev details, disabled in prod |
Variable values, loop iterations |
info |
Normal operations worth recording |
User actions, job completions, deploys |
warn |
Unexpected but handled |
Retries triggered, fallbacks activated |
error |
Failed, needs attention |
Exceptions, failed requests, timeouts |
Anti-pattern: Don't log errors for expected conditions (wrong password = info, not error).
Context Propagation
For distributed systems:
- Inherit IDs: downstream services must receive correlation IDs from upstream
- Pass through boundaries: HTTP headers, message queues, async jobs
- Middleware injection: auto-inject context into every log via middleware/interceptor
[Client] --request_id--> [API Gateway] --request_id--> [Service A] --request_id--> [Service B]
| | |
(logs) (logs) (logs)
↓ ↓ ↓
All queryable by single request_id
Async jobs: Store and restore original request context when processing background work.
What to Log
| Log These |
Skip These |
| Request entry/exit with duration |
Sensitive data (passwords, tokens, PII, cards) |
| State transitions (created → paid → shipped) |
Inside tight loops |
| External service calls with latency + status |
Success cases with no debug value |
| Auth/authz events |
Redundant infra logs (LB already captures) |
| Job starts, completions, failures |
|
| Retry attempts, circuit breaker changes |
|
Naming Conventions
| Pattern |
Example |
Field names: snake_case |
user_id, not userId or user-id |
| Events: past tense verbs |
payment_completed, not complete_payment |
| Domain prefixes when helpful |
auth.login_failed, billing.invoice_created |
Team agreement: Define field names once, use consistently across all services.
Performance
| Concern |
Solution |
| High-volume debug logs |
Sampling in production |
| Hot path logging |
Avoid or use async appenders |
| I/O overhead |
Buffer and batch writes |
| Dynamic verbosity |
Runtime-configurable log levels |
Language-Specific Implementations
| Language |
Library |
Notes |
| Python |
structlog or standard-library logging with a JSON formatter |
Verify current framework integration |
| Ruby/Rails |
Rails.event where supported, or a structured logger |
Verify the application and Rails versions |
| Node.js |
pino, winston with JSON formatter |
|
| Go |
slog (stdlib), zerolog |
|
| Java |
logback with JSON encoder |
|
Decision Table: Log or Not?
| Scenario |
Decision |
Reason |
| User enters wrong password |
info |
Expected behavior, not an error |
| Payment gateway timeout |
error + retry |
Needs attention, affects user |
| Cache miss |
debug |
Only useful for performance analysis |
| User created account |
info |
Business event worth recording |
| Loop iteration 5000 of 10000 |
Don't log |
Creates noise, no debug value |
| External API returns 500 |
warn or error |
Depends on retry/fallback behavior |
| Background job started |
info |
Useful for job debugging |
| Background job failed after retries |
error |
Needs investigation |
Incident Debugging Checklist
When designing logs, verify you can answer:
Post-incident: Add the logs you wished you had.
1---2name: structured-logging3description: Design structured application logging for observability and incident debugging. Use when adding JSON logs, correlation or trace identifiers, context propagation, log-level policy, redaction, or production debugging signals.4---56# Structured Logging78## Core Philosophy910- Logs are optimized for **querying**, not writing. Design with debugging in mind.11- A log without correlation IDs is useless in distributed systems12- If you can't answer "Who was affected? What failed? When? Why?" within 5 minutes, logging needs work1314## Structured Format1516Always use key-value pairs (JSON), never string interpolation.1718```json19{20 "event": "payment_failed",21 "user_id": "123",22 "reason": "insufficient_funds",23 "amount": 99.99,24 "timestamp": "2025-01-24T20:00:00Z",25 "level": "error",26 "service": "billing",27 "request_id": "req_abc123"28}29```3031## Required Fields3233Every log event MUST include:3435| Field | Format | Example |36|-------|--------|---------|37| `timestamp` | ISO 8601 with timezone | `2025-01-24T20:00:00Z` |38| `level` | debug, info, warn, error | `info` |39| `event` | snake_case, past tense | `user_login_succeeded` |40| `request_id` or `trace_id` | UUID or prefixed ID | `req_abc123` |41| `service` | Service/app name | `api-gateway` |42| `environment` | prod, staging, dev | `prod` |4344## High-Cardinality Fields4546Include these when available. They make logs queryable during incidents:4748| Category | Fields |49|----------|--------|50| Identity | `user_id`, `org_id`, `account_id` |51| Tracing | `request_id`, `trace_id`, `span_id` |52| Domain | `order_id`, `transaction_id`, `job_id` |5354**Rule:** Look for domain-specific identifiers that help isolate issues to specific entities.5556## Log Levels5758| Level | When to Use | Example |59|-------|-------------|---------|60| `debug` | Verbose local dev details, disabled in prod | Variable values, loop iterations |61| `info` | Normal operations worth recording | User actions, job completions, deploys |62| `warn` | Unexpected but handled | Retries triggered, fallbacks activated |63| `error` | Failed, needs attention | Exceptions, failed requests, timeouts |6465**Anti-pattern:** Don't log errors for expected conditions (wrong password = info, not error).6667## Context Propagation6869For distributed systems:70711. **Inherit IDs:** downstream services must receive correlation IDs from upstream722. **Pass through boundaries:** HTTP headers, message queues, async jobs733. **Middleware injection:** auto-inject context into every log via middleware/interceptor7475```76[Client] --request_id--> [API Gateway] --request_id--> [Service A] --request_id--> [Service B]77 | | |78 (logs) (logs) (logs)79 ↓ ↓ ↓80 All queryable by single request_id81```8283**Async jobs:** Store and restore original request context when processing background work.8485## What to Log8687| Log These | Skip These |88|-----------|------------|89| Request entry/exit with duration | Sensitive data (passwords, tokens, PII, cards) |90| State transitions (created → paid → shipped) | Inside tight loops |91| External service calls with latency + status | Success cases with no debug value |92| Auth/authz events | Redundant infra logs (LB already captures) |93| Job starts, completions, failures | |94| Retry attempts, circuit breaker changes | |9596## Naming Conventions9798| Pattern | Example |99|---------|---------|100| Field names: `snake_case` | `user_id`, not `userId` or `user-id` |101| Events: past tense verbs | `payment_completed`, not `complete_payment` |102| Domain prefixes when helpful | `auth.login_failed`, `billing.invoice_created` |103104**Team agreement:** Define field names once, use consistently across all services.105106## Performance107108| Concern | Solution |109|---------|----------|110| High-volume debug logs | Sampling in production |111| Hot path logging | Avoid or use async appenders |112| I/O overhead | Buffer and batch writes |113| Dynamic verbosity | Runtime-configurable log levels |114115## Language-Specific Implementations116117| Language | Library | Notes |118|----------|---------|-------|119| Python | `structlog` or standard-library logging with a JSON formatter | Verify current framework integration |120| Ruby/Rails | `Rails.event` where supported, or a structured logger | Verify the application and Rails versions |121| Node.js | `pino`, `winston` with JSON formatter | |122| Go | `slog` (stdlib), `zerolog` | |123| Java | `logback` with JSON encoder | |124125## Decision Table: Log or Not?126127| Scenario | Decision | Reason |128|----------|----------|--------|129| User enters wrong password | `info` | Expected behavior, not an error |130| Payment gateway timeout | `error` + retry | Needs attention, affects user |131| Cache miss | `debug` | Only useful for performance analysis |132| User created account | `info` | Business event worth recording |133| Loop iteration 5000 of 10000 | Don't log | Creates noise, no debug value |134| External API returns 500 | `warn` or `error` | Depends on retry/fallback behavior |135| Background job started | `info` | Useful for job debugging |136| Background job failed after retries | `error` | Needs investigation |137138## Incident Debugging Checklist139140When designing logs, verify you can answer:141142- [ ] **Who:** can filter to specific user/org/account?143- [ ] **What:** can identify the exact operation that failed?144- [ ] **When:** can narrow to specific time window?145- [ ] **Why:** is error context captured (reason, upstream cause)?146- [ ] **Where:** can trace across services via correlation ID?147148**Post-incident:** Add the logs you wished you had.
majesticlabs-dev