Opentelemetry Skill

OpenTelemetry Skill

Core Principles

Use these defaults:

1. Stability over Features: Check otelcol-contrib stability (Alpha/Beta/Stable) and warn on non-stable components in production.

2. Convention over Configuration: Prefer OpenTelemetry Semantic Conventions over custom attribute names.

3. Protocol Unification: Default to OTLP gRPC (4317); use OTLP HTTP (4318) when gRPC is blocked by the agent, proxy, browser, or backend.

4. Deterministic Routing Keys: Use stable routing keys for load-balancing exporters (traceID for tail sampling, tenant_id or cluster for tenant/shard routing). Normalize non-string attributes first.

5. Safety First: Prefer collector stability (memory limiters, persistent queues, backpressure) over completeness. Dropping data is better than crashing the collector.

6. Cardinality Awareness: High-cardinality attributes (>100 unique values) must not be metric dimensions; use traces or logs instead.

7. Security by Default: Redact PII, enable TLS for cross-network communication, and authenticate all collector endpoints.

8. Cross-Field Consistency: Treat collector reviews as systems reviews. Check processor order, memory limits, replica strategy, routing, metric temporality, queue storage, PDB/HPA settings, and OTTL types together before calling a config safe.

Pre-Flight Checklist

Before generating config/code, confirm these. If unknown, ask first:

1. Signal volume — High traffic (>10k RPS) or low volume? Drives sampling/scaling. → sampling.md, collector.md
2. Cardinality risk — Any unbounded metric attributes (user/request/session IDs)? Move to traces/logs. → instrumentation.md
3. Resiliency — Is restart/outage data loss acceptable? If no, use file_storage + persistent queues. → collector.md
4. Trust boundaries — Any public-network hops? Require TLS + mTLS. → security.md
5. Deployment target — Kubernetes, EC2, Lambda, or containers? → architecture.md

Eval-Critical Response Minimums

When user requests match these patterns, include these points explicitly:

• Collector setup: include memory_limiter; keep it first in each pipeline processors list; explain OOM-prevention rationale.
• Metric dimension request for user_id: refuse; explain time-series explosion risk; suggest traces and bounded metric dimensions.
• Kubernetes tail sampling: Gateway (Deployment) tier, load_balancing with routing_key: traceID, Headless Service (clusterIP: None), error+10% policies, Beta stability caution.
• Claude Code telemetry: include CLAUDE_CODE_ENABLE_TELEMETRY=1, OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative, and managed-settings persistence; traces are beta, while metrics and logs/events remain the broadly documented signals. Keep prompt/tool-content capture disabled unless PII controls are explicit, and avoid session.id as a metric dimension.
• AI agent tool-call tracing: for agents using current gen_ai.* conventions, set gen_ai.operation.name: execute_tool, preserve gen_ai.tool.name, and use the stable span name execute_tool; keep the actual tool name in the attribute.
• GenAI provider vs agent identity: preserve gen_ai.provider.name for the model/provider (for example, openai or gcp.gen_ai); use service.name or a natively emitted agent attribute for the coding-agent identity rather than writing the agent name into the provider field.

Existing Configuration Review Mode

When the user provides an existing collector config, Helm values file, or Kubernetes manifest, audit it for internal contradictions before proposing edits.

Check these interactions together:

1. Memory vs pod limit — limit_mib must leave headroom for the Go runtime and buffers.
2. Stateful processing vs scaling — tail_sampling, span_metrics, and service_graph need sticky routing above one replica.
3. Durability vs outage tolerance — disabled retries and no persistent queue mean data loss on backend failure.
4. Deployment mode vs exposure — hostPort fits DaemonSet/node-local patterns, not scaled gateway Deployments.
5. Rollout settings — review replicaCount, HPA minReplicas, PodDisruptionBudget, and rolling updates together.
6. OTTL/filter correctness — keep attribute types consistent and prefer current semantic convention keys.
7. Metric temporality and state — deltatocumulative / cumulativetodelta need source/backend/restart checks.
8. Queue storage backend — file_storage needs local locking-safe storage; RWX, EFS, and NFS are unsafe defaults.
9. Generated metric identity — for log/span-to-metric connectors, verify replica count, export mode, actual temporality, whether the emitted producer identity survives backend mapping, and the aggregation query shape together.

Progressive Disclosure: Context Triggers

Load detailed reference documentation only when the user's request matches a trigger. This keeps context lean.

Production Baseline Configuration

Use this copy-paste-ready baseline unless the user wants something different:

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"
  file_storage/queue:
    directory: /var/lib/otelcol/queue
    timeout: 10s
    compaction:
      on_start: true
      on_rebound: false

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  otlp:
    endpoint: "your-backend:4317"
    sending_queue:
      enabled: true
      storage: file_storage/queue
      num_consumers: 4
      queue_size: 1024
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 30s
      max_elapsed_time: 300s
  # otlphttp:                        # HTTP exporter — use when backend requires HTTP
  #   endpoint: "https://your-backend:4318"
  #   sending_queue: { enabled: true, storage: file_storage/queue }

service:
  extensions: [health_check, file_storage/queue]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

Key defaults:

• memory_limiter must be first in every processor chain.
• batch reduces exporter network calls.
• file_storage preserves queues across restarts only on the same host/volume. In Kubernetes, back /var/lib/otelcol/queue with a ReadWriteOnce block-backed PVC, not RWX/network storage.
• health_check binds to localhost (not 0.0.0.0) in shared networks.
• Prefer OTLP gRPC (port 4317) for receivers and exporters. Fall back to OTLP HTTP (port 4318) when gRPC is unavailable.

Validation & Error Recovery

Always include at least one validation checkpoint: otelcol validate --config <path>. For container dry-run commands, live pipeline checks, and symptom→cause→fix recovery, load validation.md.

Anti-Patterns to Avoid

❌ Placing memory_limiter anywhere except first in the processor chain ❌ Using high-cardinality attributes (user_id, trace_id) as metric dimensions ❌ Exposing pprof (1777), zpages (55679) on 0.0.0.0 in production ❌ Using tail_sampling without sticky session load balancing (load_balancing exporter) ❌ Omitting batch processor (causes excessive network calls) ❌ Calling a config "fine" because it parses, without checking memory limits, sticky routing, exporter durability, and rollout settings together

See anti-patterns.md for the full annotated catalog.

Version and Compatibility

• Use compatibility.md for fast-moving version floors and AI agent support details.
• Keep SKILL.md focused on routing logic, guardrails, and production defaults rather than inline release tracking.