Observability

Observability Rules (OpenTelemetry)

1. Core Concepts

OpenTelemetry provides a unified standard for collecting telemetry data.

Three Pillars + Context

Architecture

[Application + OTel SDK]
    |-- API (instrumentation interface)
    |-- SDK (implementation: sampling, batching, export)
    |-- Auto-instrumentation (zero-code)
         |
    [OTel Collector] (optional but recommended)
    |-- Receivers  → Processors → Exporters
         |
    [Backends: Jaeger, Tempo, Prometheus, Loki]

OTLP Protocol

Endpoints: /v1/traces, /v1/metrics, /v1/logs

2. Distributed Tracing

Span Structure

A span represents a unit of work with:

• Span Context: trace ID, span ID, trace flags (immutable)
• Attributes: key-value metadata (http.request.method, db.system)
• Events: timestamped annotations within the span
• Links: causal relationships to other spans (async flows)
• Status: Unset (default), Error, Ok

SpanKind

W3C Trace Context Propagation

traceparent: 00-<trace-id>-<span-id>-<trace-flags>
tracestate:  vendor-specific data

• Default propagator in OTel
• Inject on outgoing requests, extract on incoming
• Never propagate internal trace data to untrusted external services
• Never put sensitive data in Baggage

Instrumentation Approaches

Rule: Start with auto-instrumentation, add manual spans only for business-critical operations that auto-instrumentation doesn't cover.

3. Metrics

Instrument Types

Timer note: Some frameworks (e.g., Micrometer) provide a Timer type that combines duration measurement with count. In OTel, use a Histogram instrument for the same purpose (e.g., http.server.request.duration).

Exemplars

Link metrics to traces for drill-down from aggregated data to individual requests.

• Attach trace ID / span ID to metric measurements
• Configure with TraceBased exemplar filter
• Visualized as diamond markers in Grafana
• Enables: "This p99 latency spike → show me the exact trace"

Views

Customize metric processing per instrument:

• Select which instruments to process
• Override aggregation strategy (e.g., explicit bucket histogram)
• Filter or rename attributes
• Set aggregation temporality

Metric Naming Convention

# OTel standard: dot-separated, lowercase, include unit
http.server.request.duration
db.client.operation.duration

# Domain-specific pattern: <domain>.<entity>.<action>
orders.created.total
payments.processing.duration
users.active.count
cache.hits.total

Use standard units: s (seconds), By (bytes), {request} (count).

Key Metrics to Monitor

Application Metrics

Business Metrics

Infrastructure Metrics

4. Logs

OTel does not replace existing logging frameworks. It bridges them.

Integration Pattern

[Application Code]
    → [Logging Framework (Logback, Log4j, winston)]
        → [OTel Log Appender/Bridge]
            → [OTel SDK LogRecordProcessor]
                → [OTel Collector or Backend]

Log-Trace Correlation

When OTel SDK is active, trace ID and span ID are automatically injected into log records. No code changes required.

{
  "timestamp": "2026-03-24T10:30:00Z",
  "severity": "ERROR",
  "body": "Payment processing failed",
  "traceId": "abc123...",
  "spanId": "def456...",
  "attributes": {
    "user.id": "42",
    "order.id": "ORD-789"
  }
}

Log Rules

• Use structured logging (JSON) for machine readability
• Let OTel SDK inject trace context automatically
• Do not call the Logs Bridge API directly from application code
• Configure log appenders for your framework (Logback, Log4j2, Python logging)

5. OTel Collector

The Collector is a vendor-agnostic proxy that receives, processes, and exports telemetry data. For detailed pipeline configuration, see references/otel-collector.md.

Deployment Patterns

Recommendation: Use Agent mode in production for reliability. Gateway mode for cross-cluster aggregation and routing.

Essential Processors

6. Semantic Conventions

Use standard attribute names for interoperability across tools and dashboards. For the full convention list, see references/semantic-conventions.md.

Key Conventions (Summary)

7. SDK Patterns

For detailed setup examples by language (Java, Node.js, Python), see references/otel-sdk-patterns.md.

Quick Reference

8. Backend Integration

Recommended Stack (Grafana)

Traces  → Grafana Tempo  (OTLP native)
Metrics → Prometheus      (OTLP receiver or remote write)
Logs    → Grafana Loki    (OTLP native, Loki 3.0+)
UI      → Grafana         (unified query across all signals)

Jaeger

• Jaeger v2 uses OTel Collector as its core pipeline
• OTLP endpoints: gRPC 4317, HTTP 4318
• jaegertracing/all-in-one Docker image for dev

Prometheus OTLP

# Enable OTLP receiver
prometheus --web.enable-otlp-receiver

# OTel SDK environment variables
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:9090/api/v1/otlp/v1/metrics

9. Alerting

Design alerts around symptoms, not causes. For detailed alerting rules, severity levels, and templates, see references/alerting-rules.md.

Key Principles

• Alert on symptoms (error rate, latency), not causes (CPU, memory)
• Every alert must have a runbook or action item
• Avoid alert fatigue — only alert on actionable conditions
• Use multi-window or burn-rate alerts over simple thresholds
• Include context in alert messages (service, environment, metric value)

Alert Severity Summary

10. Health Checks

Health checks enable orchestrators (Kubernetes, load balancers) to manage application lifecycle. For detailed probe configuration and rules, see references/health-checks.md.

Probe Types

Key Rules

• Liveness probes must be lightweight — no external dependency checks
• Readiness probes should verify critical dependencies
• Never put slow checks in liveness probes (causes unnecessary restarts)
• Health checks must not cause side effects (writes, external calls)

11. Common Anti-Patterns

12. SLO/SLI Design

Service Level Objectives (SLOs) and Indicators (SLIs) translate reliability into measurable targets. For detailed design patterns, see references/slo-sli-design.md.

SLI Selection Framework

SLO Design Rules

• Set SLO targets based on user pain thresholds, not current performance — aspire and improve
• Define error budget = (1 - SLO) × time window; consume it deliberately on features, not incidents
• Use multi-window multi-burn-rate alerts (5m + 1h short window, 30m + 6h long window)
• Review and adjust SLOs quarterly — SLOs should reflect current user expectations
• Start conservative (e.g., 99.0%) and tighten as reliability improves

Error Budget Policy

13. Related Skills

• logging: Structured logging and monitoring integration
• incident-response: Alert-driven incident response processes
• troubleshooting: Monitoring data-driven problem diagnosis
• spring-framework: Spring Boot Actuator and Micrometer metrics

Additional References

• For PromQL examples, recording rules, and common query patterns, see references/prometheus-queries.md
• For SLO/SLI design, error budget management, and multi-window burn rate alerting, see references/slo-sli-design.md
• OpenTelemetry Documentation - Official OpenTelemetry documentation
• Prometheus Documentation - Official Prometheus documentation
• Google SRE Book - Monitoring - Monitoring distributed systems
• For Spring Boot implementation patterns (Actuator, Micrometer, distributed tracing), see spring-framework skill — references/monitoring.md