Opentelemetry

OpenTelemetry

Vendor-neutral observability framework for distributed tracing, metrics, and logs. Provides SDKs for application instrumentation and a Collector for receiving, processing, and exporting telemetry data.

Architecture

App (SDK) → OTEL Collector → Backend (Jaeger / Tempo+Grafana)
    ↓              ↓
  Traces       Pipelines:
  Metrics      receivers → processors → exporters
  Logs

Task Reference

Go Instrumentation

• SDK setup, TracerProvider, SpanProcessor → references/languages/go.md
• Creating spans, attributes, events, status → references/languages/go.md
• HTTP/gRPC middleware, database tracing → references/languages/go.md

Trace Backends

• Jaeger deployment, UI, query patterns → references/backends/jaeger.md
• Tempo + Grafana + TraceQL queries → references/backends/tempo-grafana.md

Observability Pillars

• Spans, context propagation, sampling, perf analysis → references/observability/tracing.md
• Metrics SDK, custom metrics, exemplars → references/observability/metrics.md
• Structured logging with trace/span IDs → references/observability/log-correlation.md

Framework Instrumentation

• Django auto/manual instrumentation, ORM tracing → references/frameworks/django.md
• FastAPI instrumentation, SQLAlchemy, async patterns → references/frameworks/fastapi.md
• Go frameworks: Gin, Echo, Chi, Fiber middleware → references/frameworks/gin.md
• Node.js: Express, NestJS, Fastify, Prisma → references/frameworks/express.md

Collector Infrastructure

• OTEL Collector on Kubernetes (DaemonSet/Sidecar) → references/collector/setup.md
• Pipeline config (receivers, processors, exporters) → references/collector/setup.md

Quick Start (Go)

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("my-service"),
        )),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.TraceContext{})
    return tp, nil
}

Identifying Slowdowns

1. Add spans around suspected slow operations (DB queries, HTTP calls, processing loops)
2. Add timing attributes — span.SetAttributes(attribute.Int64("db.rows", count))
3. Query in Jaeger/Tempo — filter by duration > 500ms or sort by duration
4. Drill into trace waterfall — identify which child span dominates total latency
5. Use exemplars — link slow metric samples directly to the trace that caused them

See references/observability/tracing.md for detailed span analysis patterns.

Key Go Packages

Key Python Packages

Key Node.js Packages

Common Mistakes

Official Documentation

• OpenTelemetry Go SDK
• OpenTelemetry Python SDK
• OpenTelemetry JS SDK
• OTEL Collector
• Semantic Conventions

Related skill: For Grafana dashboards, Prometheus, and ServiceMonitors — see the grafana skill.

Source: kettleofketchup/KettleOfSkills — distributed by TomeVault.