Golang O11y

Go Observability (OpenTelemetry + Prometheus + slog)

Observability in Go rests on three pillars: traces (request flow), metrics (counters/gauges/histograms), and logs (structured events). Correlate them with Trace ID, Span ID, and Request ID.

When to Use This Skill

• Adding or fixing observability in Go microservices
• Setting up distributed tracing (OpenTelemetry) across services
• Exposing Prometheus metrics and /metrics endpoint
• Using slog for structured, trace-correlated logging
• Implementing health checks (liveness/readiness) and graceful shutdown
• Debugging production issues with traces and metrics

Core Principles

• Traces: Request flow across services; use context propagation.
• Metrics: Counters, gauges, histograms; keep label cardinality bounded.
• Logs: slog (Go 1.21+) with trace_id/span_id for correlation.

All three should share Trace ID (and optionally Request ID) so logs and metrics can be tied to spans.

Quick Start

1. Tracer: Initialize OTel tracer provider with OTLP or Jaeger exporter; set as global provider; defer tp.Shutdown(ctx).
2. Spans: ctx, span := otel.Tracer("svc").Start(ctx, "op"); defer span.End(); pass ctx through the call chain.
3. HTTP: Use otelhttp.NewHandler(handler, "route-name") for auto-instrumentation.
4. Metrics: Register Prometheus counters/histograms; expose /metrics with promhttp.Handler().
5. Logs: Use slog with slog.With("trace_id", traceID); include trace_id in log attributes.
6. Health: Implement /health (liveness) and /ready (readiness); call tp.Shutdown on SIGTERM.

Quick Reference

// Init tracer
tp, _ := initTracer("service-name")
defer tp.Shutdown(context.Background())

// Span
ctx, span := otel.Tracer("name").Start(ctx, "operation")
defer span.End()
span.SetAttributes(attribute.String("key", "value"))

// Metrics
counter := promauto.NewCounterVec(opts, []string{"label"})
histogram := promauto.NewHistogramVec(opts, []string{"label"})

// Logging
logger := slog.With("trace_id", traceID)
logger.Info("message", "key", value)

// Health
http.HandleFunc("/health", livenessHandler)
http.HandleFunc("/ready", readinessHandler)

Additional Resources

• Full reference: Setup, code examples, decision trees (OTel vs Prometheus vs slog), anti-patterns, metric naming — see reference.md.
• Official: OpenTelemetry Go, Prometheus client_golang, log/slog.

Source: Marz32onE/otel-traces-test — distributed by TomeVault.