Opentelemetry

OpenTelemetry Observability AI Skill Guide

Overview

OpenTelemetry (OTel) is a vendor-neutral standard for traces, metrics, and logs. Applications use language SDKs/APIs; the OpenTelemetry Collector receives, processes, and exports telemetry to backends (Jaeger, Prometheus, cloud APM). Agents should prioritize correct context propagation and low-cardinality attributes over dumping every field into spans.

App SDK (traces/metrics/logs)
        |
        | OTLP
        v
OTel Collector (receivers -> processors -> exporters)
        |
        +--> Tempo/Jaeger (traces)
        +--> Prometheus/Mimir (metrics)
        +--> Loki/ELK (logs)

When to use

• Adding instrumentation to HTTP/gRPC clients and servers
• Designing Collector pipelines and sampling strategies
• Debugging missing spans, broken trace IDs, or cardinality explosions
• Standardizing attributes (service.name, deployment.environment)

Operational directives

1. Always set service.name (and ideally service.version) via resource attributes.
2. Propagate W3C Trace Context (traceparent) across process boundaries.
3. Prefer semantic conventions for HTTP, DB, messaging attributes.
4. Sample thoughtfully in production - head sampling or tail sampling in Collector.
5. Avoid high-cardinality labels (raw user IDs, unbounded URLs) on metrics.

Concrete examples

Collector pipeline sketch

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch: {}
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }
  prometheus:
    endpoint: 0.0.0.0:8889
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

Node.js manual span sketch

const { trace } = require("@opentelemetry/api");
const tracer = trace.getTracer("api");

async function createOrder(req) {
  return tracer.startActiveSpan("createOrder", async (span) => {
    span.setAttribute("order.items", req.items.length);
    try {
      const id = await saveOrder(req);
      span.setAttribute("order.id", id);
      return id;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: 2, message: String(err) });
      throw err;
    } finally {
      span.end();
    }
  });
}

Propagation check

# Expect traceparent on outbound calls; verify backend shows one trace ID
curl -sI https://api.example.com/health | tr -d '\r' | grep -i trace

Common failure modes

Best practices

1. Auto-instrument frameworks first; add manual spans at business boundaries.
2. Keep Collector config in Git; use processors for redaction (PII scrubbing).
3. Correlate logs with trace_id / span_id fields.
4. Define SLOs from RED/USE metrics derived from OTel, not ad-hoc counters only.

Limitations

• Spec and semantic conventions evolve - pin SDK versions.
• Logs signal maturity varies by language; confirm SDK support.
• OTel does not replace on-call process or runbooks.

Related skills

• kubernetes - sidecars/DaemonSets for Collector deployment
• elasticsearch - log backend often paired with traces
• docker - local Collector + app compose stacks
• nginx-hardening - edge headers and tracing passthrough