Opentelemetry

related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd

OpenTelemetry in Cloud-Native Engineering

Category: Observability
Status: Active
Stars: 9,100
Last Updated: 2026-04-22
Primary Language: Go
Documentation: Observability framework for tracing, metrics, and logs with vendor-neutral APIs

Purpose and Use Cases

OpenTelemetry is a core component of the cloud-native ecosystem, serving as vendor-neutral APIs

What Problem Does It Solve?

OpenTelemetry addresses the challenge of distributed tracing and observability for cloud-native applications. It provides standardized telemetry collection, vendor-neutral APIs, and comprehensive ecosystem.

When to Use This Project

Use OpenTelemetry when monitoring microservices, debugging distributed systems, or requiring detailed performance insights. Not ideal for simple deployments or when need consistent telemetry across services, want to avoid vendor lock-in, or require detailed tracing.

Key Use Cases

• Distributed Tracing for Microservices
• Application Performance Monitoring
• Log Correlation and Analysis
• Infrastructure Observability
• Business Analytics via Telemetry

Architecture Design Patterns

Core Components

• APIs: Language-specific APIs for instrumenting code
• SDKs: Implementation of telemetry collection
• Collector: Receives, processes, and exports telemetry data
• Exporters: Send telemetry to various backends
• Importers: Convert telemetry from other formats

Component Interactions

1. Application → SDK: Application creates spans and metrics via SDK
2. SDK → Collector: SDK exports telemetry to collector
3. Collector → Backend: Collector processes and exports to observability backends
4. Backend → Dashboard: Data visualized in dashboards and alerts

Data Flow Patterns

1. Trace Propagation: Span context propagated via headers across service boundaries
2. Metric Collection: Metrics collected, aggregated, and exported
3. Log Enrichment: Logs enhanced with trace context
4. Data Processing: Collector applies processors before export

Design Principles

• Standardization: Open standards via CNCF specification
• Backward Compatibility: Support existing instrumentation
• Performance: Low overhead design
• Vendor Neutrality: No lock-in to specific backend

Integration Approaches

Integration with Other CNCF Projects

• Prometheus: Metrics collection integration
• Jaeger: Tracing backend
• Zipkin: Alternative tracing backend
• OTLP: OpenTelemetry Protocol for data exchange

API Patterns

• Tracer API: Create and manage spans
• Meter API: Create and record metrics
• Logger API: Create and enrich logs
• Propagators: Trace context propagation

Configuration Patterns

• YAML Configuration: Collector configuration
• Environment Variables: Runtime configuration
• Helm Chart: Kubernetes deployment

Extension Mechanisms

• Custom Exporters: Send telemetry to proprietary backends
• Custom Processors: Transform telemetry data
• Custom Receivers: Receive telemetry from sources

Common Pitfalls and How to Avoid Them

Misconfigurations

• High Cardinality: Too many unique metric labels
  • How to Avoid: Use cardinality limits, review label values
• Trace Context Loss: Trace IDs not propagated between services
  • How to Avoid: Ensure propagators configured, check header propagation

Performance Issues

• Instrumentation Gaps: Missing coverage in code
  • How to Avoid: Use auto-instrumentation, review coverage
• Security Exposure: Sensitive data in spans
  • How to Avoid: Implement redaction, use secure exporters

Operational Challenges

• Configuration Drift: Inconsistent configs across services
  • How to Avoid: Centralize configuration, use version control
• Network Bottlenecks: Collector network saturation
  • How to Avoid: Scale collectors, use aggregation

Security Pitfalls

Coding Practices

Idiomatic Configuration

• Span Management: Create spans for meaningful operations
• Attribute Naming: Use consistent attribute naming conventions
• Sampling Configuration: Configure sampling rates appropriately

API Usage Patterns

• Manual Instrumentation: Add spans programmatically
• Auto-Instrumentation: Use agent-based instrumentation
• Context Propagation: Propagate trace context in headers

Observability Best Practices

• Metric Aggregation: Pre-aggregate metrics where possible
• Span Attributes: Include relevant contextual attributes
• Log Correlation: Include trace IDs in logs

Testing Strategies

• Unit Tests: Test instrumentation logic
• Integration Tests: Test end-to-end telemetry flow
• Performance Tests: Validate overhead levels

Development Workflow

• Local Development: Use local collector and backend
• Debug Commands: Use otel-cli for testing
• Test Environment: Set up isolated observability stack
• CI/CD Integration: Automate telemetry testing
• Monitoring Setup: Configure alerting on metrics
• Documentation: Maintain instrumentation guides

Fundamentals

Essential Concepts

• Trace: Request flow through services
• Span: Individual operation within a trace
• Span Context: Trace state propagated between services
• Span Kind: Types: SERVER, CLIENT, PRODUCER, CONSUMER
• Metric: Numerical data point
• Histogram: Distribution of values
• Gauge: Single numerical value
• Counter: Monotonically increasing metric
• Logger: Log record with attributes
• Resource: Entity producing telemetry
• Scope: Instrumentation library
• Exporter: Sends telemetry to backend
• Processor: Transforms telemetry data
• Collector: Receives and processes telemetry

Terminology Glossary

• OTLP: OpenTelemetry Protocol
• SDK: Software Development Kit
• API: Application Programming Interface
• Sampler: Decides if spans are sampled
• ** Propagator**: Handles context propagation
• Exporter: Sends data to backend

Data Models and Types

• SpanData: Span information and attributes
• MetricData: Metric points and aggregations
• LogRecord: Log entry with attributes
• ResourceData: Resource attributes
• ScopeData: Instrumentation scope

Lifecycle Management

• Trace Lifecycle: Start trace → Create spans → Propagate → End
• Metric Lifecycle: Create instrument → Record points → Export
• Span Lifecycle: Start → Add events → Set status → End
• Exporter Lifecycle: Receive → Batch → Export → Confirm

State Management

• Trace State: Active traces in memory
• Span State: Current span lifecycle state
• Metric State: Aggregated metric values
• Exporter State: Pending and exported data

Scaling and Deployment Patterns

Horizontal Scaling

• Collector Scaling: Scale collectors horizontally
• Export Rate Limiting: Limit export rate per backend
• Sampler Scaling: Adjust sampling based on load

High Availability

• Collector HA: Multiple collector instances
• Batch Processing: Batch spans before export
• Retry Logic: Retry failed exports
• Load Balancing: Distribute telemetry across collectors

Production Deployments

• Collector Deployment: Deploy as sidecar or standalone
• Security Configuration: Configure TLS and authentication
• Resource Limits: Set appropriate CPU and memory limits
• Monitoring Setup: Monitor collector health
• Backup Strategy: No backup needed - telemetry is ephemeral
• Logging Setup: Configure collector logs
• Alerting: Set alerts for collector issues
• Performance Tuning: Optimize buffer sizes and batch settings

Upgrade Strategies

• Chart Upgrade: Upgrade to new OpenTelemetry Collector version
• API Compatibility: Verify SDK compatibility
• Data Migration: No migration needed
• Testing: Validate telemetry still flows

Resource Management

• CPU Resources: Set collector CPU limits
• Memory Resources: Configure collector memory limits
• Storage Resources: No persistent storage needed
• Network Resources: Configure egress rules

Additional Resources

• Official Documentation: https://opentelemetry.io/docs/
• GitHub Repository: Check the project's official documentation for repository link
• CNCF Project Page: cncf.io/projects/cncf-opentelemetry/
• Community: Check the official documentation for community channels
• Versioning: Refer to project's release notes for version-specific features

Troubleshooting

Common Issues

1. Deployment Failures

   • Check pod logs for errors
   • Verify configuration values
   • Ensure network connectivity

2. Performance Issues

   • Monitor resource usage
   • Adjust resource limits
   • Check for bottlenecks

3. Configuration Errors

   • Validate YAML syntax
   • Check required fields
   • Verify environment-specific settings

4. Integration Problems

   • Verify API compatibility
   • Check dependency versions
   • Review integration documentation

Getting Help

• Check official documentation
• Search GitHub issues
• Join community channels
• Review logs and metrics Content generated automatically. Verify against official documentation before production use.

Examples

Basic Configuration

# Basic configuration example
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{project_name}}-config
  namespace: default
data:
  # Configuration goes here
  config.yaml: |
    # Base configuration
    # Add your settings here

Kubernetes Deployment

# Kubernetes deployment for {{project_name}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{project_name}}
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{project_name}}
  template:
    metadata:
      labels:
        app: {{project_name}}
    spec:
      containers:
      - name: {{project_name}}
        image: {{project_name}}:latest
        ports:
        - containerPort: 8080
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"

Kubernetes Service

# Kubernetes service for {{project_name}}
apiVersion: v1
kind: Service
metadata:
  name: {{project_name}}
  namespace: default
spec:
  selector:
    app: {{project_name}}
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: ClusterIP

When to Use

Use this skill when:

• Integrating a CNCF project into Kubernetes infrastructure — You need to configure, deploy, or troubleshoot a cloud-native tool within a cluster
• Designing cloud-native architecture — You are selecting and integrating CNCF tools to solve specific infrastructure challenges
• Resolving operational issues — A CNCF component is misbehaving, underperforming, or needs configuration changes

Core Workflow

1. Assess Requirements — Understand the use case, scale, integration needs, and existing infrastructure. Checkpoint: Document requirements, constraints, and success criteria.

2. Design Architecture — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. Checkpoint: Verify the architecture addresses all requirements and follows CNCF conventions.

3. Implement & Configure — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. Checkpoint: Validate all YAML against schema and test in a staging environment.

4. Deploy & Monitor — Apply manifests to the cluster, verify component health, and confirm observability is working. Checkpoint: Confirm all pods/services are running, probes passing, and metrics/alerts configured.

Constraints

MUST DO

• Include at least one complete working YAML manifest example
• Note when content is auto-generated vs. manually verified
• Reference relevant CNCF project documentation

MUST NOT DO

• Deploy manifests without testing in a staging environment first
• Use deprecated API versions (e.g., apps/v1beta1)
• Omit resource limits and requests in Kubernetes manifests