opensearch-project Set Up Trace Collection Pipeline
You are helping the user configure a pipeline to get their agent's OTel traces into OpenSearch where Agent Health can read them.
Architecture Options
Agent → OTLP/HTTP → [Pipeline] → OpenSearch
Three pipeline options:
| Option | Best For | Complexity |
|---|---|---|
| OSIS (OpenSearch Ingestion) | AWS-managed, production | Low (managed) |
| OTel Collector | Self-hosted, flexible routing | Medium |
| Direct export | Dev/testing only | Lowest |
Option 1: OSIS Pipeline (Recommended for AWS)
Prerequisites
- AWS account with OSIS access
- OpenSearch domain (managed or serverless)
Pipeline Configuration
version: "2"
otel-trace-pipeline:
source:
otel_trace_source:
path: "/v1/traces"
processor:
- otel_trace_raw:
sink:
- opensearch:
hosts: ["https://your-opensearch-domain.us-west-2.es.amazonaws.com"]
index: "otel-v1-apm-span-*"
aws:
sts_role_arn: "arn:aws:iam::ACCOUNT:role/osis-role"
region: "us-west-2"
Agent Configuration
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-osis-pipeline.us-west-2.osis.amazonaws.com/v1/traces
OTEL_SERVICE_NAME=my-agent
Option 2: OTel Collector (Self-hosted)
docker-compose.yml
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
ports:
- "4318:4318" # OTLP HTTP
volumes:
- ./otel-config.yaml:/etc/otelcol-contrib/config.yaml
otel-config.yaml
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 5s
send_batch_size: 100
exporters:
opensearch:
http:
endpoint: "https://your-opensearch:9200"
tls:
insecure: false
dataset: "traces"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [opensearch]
Agent Configuration
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces
OTEL_SERVICE_NAME=my-agent
Option 3: Direct Export (Dev Only)
Export directly to OpenSearch Data Prepper format. Not recommended for production (no batching, no retry).
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-opensearch:9200/_plugins/_trace_analytics/v1/traces
Agent Health Configuration
Once traces are flowing to OpenSearch, configure Agent Health to read them:
# In your Agent Health .env
OPENSEARCH_LOGS_ENDPOINT=https://your-opensearch:9200
OPENSEARCH_LOGS_USERNAME=admin
OPENSEARCH_LOGS_PASSWORD=admin
# Or for AWS SigV4:
# OPENSEARCH_LOGS_AUTH_TYPE=sigv4
# OPENSEARCH_LOGS_AWS_REGION=us-west-2
Verification Checklist
- Agent sends spans: Check agent logs for "telemetry enabled" message
- Pipeline receives: Check OSIS/Collector metrics for incoming spans
- OpenSearch has data: Query
otel-v1-apm-span-*index - Agent Health sees traces: Open Traces tab, verify spans appear with correct categorization
Troubleshooting
| Issue | Check |
|---|---|
| No spans arriving | Is OTEL_EXPORTER_OTLP_ENDPOINT correct? Is the agent calling provider.register()? |
| Spans in OS but not in Agent Health | Check index pattern matches what Agent Health queries (otel-v1-apm-span-*) |
| Auth errors | Check IAM role for OSIS, or username/password for basic auth |
| Spans arrive but incomplete | Ensure provider.shutdown() is called on agent exit (flushes batch) |