Monitoring Ops

Monitoring Operations

Comprehensive observability patterns covering the three pillars (metrics, logging, tracing), alerting strategies, dashboard design, and infrastructure monitoring for production systems.

Three Pillars Quick Reference

Use this table to decide which observability signal fits your need:

When to use which:

• "How many requests per second?" → Metrics (counter + rate)
• "Why did this specific request fail?" → Logs (error message + stack trace)
• "Where is the latency in this request?" → Traces (span waterfall)
• "Is the system healthy right now?" → Metrics (gauges + alerts)
• "What happened at 3:42 AM?" → Logs (timestamped event search)
• "Which downstream service caused the timeout?" → Traces (span analysis)

Correlation is key: Connect all three by embedding trace_id in log entries, recording exemplars in metrics, and linking trace spans to log queries.

Metrics Type Decision Tree

Use this tree to select the correct metric type:

What are you measuring?
│
├─ A count of events that only goes up?
│  └─ COUNTER
│     Examples: http_requests_total, errors_total, bytes_sent_total
│     Use rate() or increase() to get per-second or per-interval values
│     Never use a counter's raw value — it resets on restart
│
├─ A current value that goes up AND down?
│  └─ GAUGE
│     Examples: temperature_celsius, active_connections, queue_depth
│     Use for snapshots of current state
│     Can use avg_over_time(), max_over_time() for trends
│
├─ A distribution of values (latency, size)?
│  │
│  ├─ Need aggregatable quantiles across instances?
│  │  └─ HISTOGRAM
│  │     Examples: http_request_duration_seconds, response_size_bytes
│  │     Define buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
│  │     Use histogram_quantile() for percentiles (p50, p95, p99)
│  │     Aggregatable across instances (histograms can be summed)
│  │
│  └─ Need pre-calculated quantiles on a single instance?
│     └─ SUMMARY
│        Examples: go_gc_duration_seconds
│        Pre-calculates quantiles client-side
│        NOT aggregatable across instances
│        Prefer histogram unless you have a specific reason
│
└─ None of the above?
   └─ INFO metric (labels only, value=1)
      Examples: build_info{version="1.2.3", commit="abc123"}
      Use for metadata exposed as metrics

Rule of thumb: Start with counters and histograms. Add gauges for current state. Avoid summaries unless you have a compelling reason.

Alerting Decision Tree

What type of alert do you need?
│
├─ Known threshold with a fixed boundary?
│  └─ THRESHOLD-BASED
│     Example: CPU > 90% for 5 minutes
│     Pros: Simple, predictable, easy to understand
│     Cons: Requires manual tuning, doesn't adapt to patterns
│     Best for: Resource limits, error rate spikes, queue depth
│
├─ Normal behavior varies by time/season?
│  └─ ANOMALY-BASED
│     Example: Traffic 3 standard deviations below normal for this hour
│     Pros: Adapts to patterns, catches novel failures
│     Cons: Noisy during transitions, requires training data
│     Best for: Traffic patterns, business metrics, gradual degradation
│
└─ Defined reliability targets?
   └─ SLO-BASED (PREFERRED)
      Example: Error budget burn rate > 14.4x for 1 hour
      Pros: Aligned with user impact, reduces noise, principled
      Cons: Requires SLI/SLO definition, more complex setup
      Best for: User-facing services, platform reliability

Severity Levels

When to Page vs When to Ticket

Page (wake someone up) when:

• Users are currently impacted
• Data loss is occurring or imminent
• Security incident is active
• Error budget will be exhausted within hours

Create ticket (don't page) when:

• Issue is not user-facing yet
• Automated remediation is possible
• Degradation is slow and has runway
• Issue is during business hours and can be triaged normally

Structured Logging Quick Reference

Standard JSON Log Format

{
  "timestamp": "2026-03-09T14:32:01.123Z",
  "level": "ERROR",
  "message": "Failed to process payment",
  "service": "payment-api",
  "version": "1.4.2",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "request_id": "req-abc123",
  "user_id": "usr-789",
  "error": {
    "type": "PaymentGatewayTimeout",
    "message": "Gateway response timeout after 30s",
    "stack": "..."
  },
  "duration_ms": 30042,
  "http": {
    "method": "POST",
    "path": "/api/v1/payments",
    "status_code": 504
  }
}

Log Level Decision Guide

Rules:

• Never log at ERROR for expected conditions (user input validation → WARN)
• Every ERROR should be actionable — if no one will act on it, use WARN
• DEBUG should be off in production by default
• INFO should not be noisy — 1-5 log lines per request, not 50

Correlation IDs

• Generate a request_id (UUID v4 or ULID) at the edge/gateway
• Propagate through all internal services via headers (X-Request-ID)
• Include trace_id and span_id from distributed tracing
• Log all three IDs in every log entry for cross-referencing

Distributed Tracing Quick Reference

Core Concepts

• Trace: End-to-end journey of a request across all services
• Span: A single unit of work (HTTP call, DB query, function execution)
• Context propagation: Passing trace/span IDs between services via headers

W3C TraceContext Header

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │  │                                  │                  │
              │  │                                  │                  └─ flags (01=sampled)
              │  │                                  └─ parent span ID (16 hex)
              │  └─ trace ID (32 hex)
              └─ version (00)

Sampling Strategies

Recommendation: Use parent-based + tail-based at the Collector. This captures all error traces and slow traces while controlling volume.

Trace ID in Logs

Always include trace_id in structured log entries. This enables jumping from a log line to the full trace view:

Log entry → trace_id → Jaeger/Tempo → full request waterfall

Tool Selection Matrix

Recommendation path:

• Starting out / budget-conscious: Prometheus + Grafana + Loki + Tempo (all free, self-hosted)
• Small team, want managed: Grafana Cloud free tier (10k metrics, 50GB logs, 50GB traces)
• Enterprise, need everything: Datadog (expensive but comprehensive)
• AWS-only shop: CloudWatch + X-Ray (simplest if already on AWS)

Dashboard Design

USE Method (Infrastructure)

For every resource (CPU, memory, disk, network):

RED Method (Services)

For every service endpoint:

Four Golden Signals (Google SRE)

Dashboard Layout Best Practices

1. Top row: Key health indicators (error rate, latency p99, availability %)
2. Second row: Traffic and throughput (requests/sec, active users)
3. Third row: Resource utilization (CPU, memory, disk, network)
4. Bottom rows: Detailed breakdowns (by endpoint, by status code, by region)
5. Use variables: Service, environment, time range as dropdown selectors
6. Include annotations: Deployments, incidents, config changes as vertical markers

Common Gotchas

Reference Files

See Also

• docker-ops — Container monitoring with cAdvisor, Docker stats, and health checks
• ci-cd-ops — Pipeline observability, deployment tracking, build metrics
• nginx-ops — Nginx access/error log parsing, request metrics, upstream monitoring
• python-observability-ops — Python-specific instrumentation with structlog, opentelemetry-python
• OpenTelemetry documentation
• Prometheus best practices
• Google SRE Book — Monitoring chapter
• Grafana dashboards library