395 Opentelemetry 77a1d317

OpenTelemetry Integration Reference

Overview

Grafana Loki supports two methods for OpenTelemetry log ingestion:

1. Native OTLP Endpoint (Recommended - Loki 3.0+)
2. LokiExporter (Deprecated)

Native OTLP Integration (Recommended)

Key Benefits

• Log body stored as plain text (not JSON encoded)
• 17 default resource attributes auto-indexed as labels
• Structured metadata for non-indexed attributes
• Simpler queries without JSON parsing
• Better storage efficiency
• All future enhancements focus on this method

Loki Configuration

loki:
  limits_config:
    # Required for OTLP (default in Loki 3.0+)
    allow_structured_metadata: true

    # Optional: Customize OTLP attribute mapping
    otlp_config:
      resource_attributes:
        attributes_config:
          # Promote additional attributes to index labels
          - action: index_label
            attributes:
              - custom.attribute
          # Drop sensitive attributes
          - action: drop
            attributes:
              - sensitive.field

OpenTelemetry Collector Configuration

Basic Configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlphttp:
    endpoint: http://loki-gateway:3100/otlp

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlphttp]

With Authentication:

extensions:
  basicauth:
    client_auth:
      username: ${LOKI_USERNAME}
      password: ${LOKI_PASSWORD}

exporters:
  otlphttp:
    endpoint: http://loki-gateway:3100/otlp
    auth:
      authenticator: basicauth

service:
  extensions: [basicauth]
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlphttp]

With Multi-Tenancy:

exporters:
  otlphttp:
    endpoint: http://loki-gateway:3100/otlp
    headers:
      X-Scope-OrgID: "my-tenant"

Resource Attribute Mapping

Loki automatically indexes these default resource attributes as labels:

Transformation Rules:

• Dots converted to underscores: service.name → service_name
• Nested attributes flattened: http.request.body → http_request_body
• Non-string values stringified automatically

OTLP Data Model

LogRecord Structure:

LogRecord:
  Timestamp            # Event occurrence time
  ObservedTimestamp    # System detection time
  TraceContext:
    TraceId            # Links to distributed trace
    SpanId             # Links to operation span
    TraceFlags         # Sampling info
  SeverityNumber       # 1-24 scale
  SeverityText         # TRACE, DEBUG, INFO, WARN, ERROR, FATAL
  Body                 # Log message
  Resource             # Source metadata
  InstrumentationScope # Emitting library info
  Attributes           # Custom key-value pairs

Severity Level Mapping:

Querying OTLP Logs

Direct Attribute Access:

# Filter by severity
{service_name="api"} | severity_text="ERROR"

# Filter by trace context
{service_name="api"} | trace_id="abc123"

# Access structured metadata
{service_name="api"} | user_id="12345"

Compare with LokiExporter (legacy):

# OTLP Native (simple)
{service_name="api"} | severity_text="ERROR"

# LokiExporter (complex - requires parsing)
{job="my-namespace/api"} | json | severity="ERROR"

LokiExporter (Deprecated)

Status: No longer recommended. No new feature development.

Why Deprecated

• All data encoded into JSON blobs
• Requires query-time JSON parsing
• Fixed index labels only: job, instance, exporter, level
• Higher query overhead
• Inefficient storage

Migration to Native OTLP

Step 1: Update Loki Configuration

loki:
  limits_config:
    allow_structured_metadata: true

Step 2: Update Collector Configuration

# Old (LokiExporter)
exporters:
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    labels:
      attributes:
        severity: ""

# New (Native OTLP)
exporters:
  otlphttp:
    endpoint: http://loki:3100/otlp

Step 3: Update LogQL Queries

# Old (LokiExporter)
{job="namespace/service"} | json | level="error"

# New (Native OTLP)
{service_name="service", service_namespace="namespace"} | severity_text="ERROR"

Grafana Alloy Configuration

Grafana Alloy is the recommended collector for Loki.

Basic OTLP to Loki:

otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }

  output {
    logs = [otelcol.exporter.otlphttp.loki.input]
  }
}

otelcol.exporter.otlphttp "loki" {
  client {
    endpoint = "http://loki-gateway:3100/otlp"
    headers = {
      "X-Scope-OrgID" = "default",
    }
  }
}

With Kubernetes Attributes:

otelcol.processor.k8sattributes "default" {
  extract {
    metadata = [
      "k8s.namespace.name",
      "k8s.pod.name",
      "k8s.deployment.name",
      "k8s.node.name",
    ]
  }

  output {
    logs = [otelcol.exporter.otlphttp.loki.input]
  }
}

Application SDK Configuration

Java (Log4j2)

<!-- log4j2.xml -->
<Configuration>
  <Appenders>
    <OpenTelemetry name="OpenTelemetryAppender"/>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="OpenTelemetryAppender"/>
    </Root>
  </Loggers>
</Configuration>

# application.properties
otel.exporter.otlp.endpoint=http://collector:4317
otel.service.name=my-java-app

Python

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

# Setup logging
logger_provider = LoggerProvider()
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(endpoint="http://collector:4317"))
)

Go

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
    "go.opentelemetry.io/otel/sdk/log"
)

func initLogger() {
    exporter, _ := otlploggrpc.New(ctx,
        otlploggrpc.WithEndpoint("collector:4317"),
        otlploggrpc.WithInsecure(),
    )

    provider := log.NewLoggerProvider(
        log.WithProcessor(log.NewBatchProcessor(exporter)),
    )
}

Node.js

const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
const { LoggerProvider } = require('@opentelemetry/sdk-logs');

const loggerProvider = new LoggerProvider();
loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(
    new OTLPLogExporter({
      url: 'http://collector:4317',
    })
  )
);

Best Practices

Resource Attribute Strategy

1. Use Semantic Conventions

   • Use standard OpenTelemetry semantic conventions
   • Consistent naming across services

2. Avoid High Cardinality

   • Don't use request IDs, user IDs as resource attributes
   • Store high-cardinality data in log attributes (structured metadata)

3. Kubernetes-Native Attributes

   • Let collector auto-detect k8s attributes
   • Use k8sattributes processor in collector

Query Optimization

1. Index Labels First

   # Good - uses indexed labels
   {service_name="api", k8s_namespace_name="prod"} | severity_text="ERROR"
   
   # Bad - no index filter
   {} | severity_text="ERROR" | service_name="api"
   

2. Use Structured Metadata for Secondary Filters

   {service_name="api"} | user_id="12345"
   

Trace Correlation

# Find logs for a specific trace
{service_name="api"} | trace_id="abc123def456"

# Link to Grafana Tempo
# Use trace_id to navigate from logs to traces

Troubleshooting

OTLP Payloads Rejected

Error: malformed request or structured metadata errors

Solution:

loki:
  limits_config:
    allow_structured_metadata: true

Missing Attributes in Labels

Issue: Resource attributes not appearing as index labels

Check:

1. Verify attribute is in default list or custom config
2. Check attribute naming follows conventions
3. Verify collector is sending attributes

High Cardinality Warnings

Issue: Too many unique label values

Solution:

1. Move high-cardinality attributes to structured metadata
2. Use otlp_config to drop or not index certain attributes

Connection Issues

# Test collector to Loki connectivity
curl -v http://loki-gateway:3100/ready

# Check collector logs
kubectl logs -l app=otel-collector -c collector

# Verify endpoint format
# Correct: http://loki:3100/otlp
# Wrong: http://loki:3100/loki/api/v1/push (that's the push API)