Otel

OpenTelemetry (Python-first)

The single reference for OTel in this project. Replaces the four upstream dash0 skills. Vendor-neutral — no backend-specific guidance.

Layered above python-infrastructure → references/observability.md, which pins the project-specific conventions (resource attributes, log severity, golden signals, NATS context propagation). Read that first if you're applying this in code; come here for the deep reference.

Scope routing

Key principles

1. Auto-instrument first, manual for business operations. opentelemetry-bootstrap -a install covers FastAPI, httpx, asyncpg, redis, requests, SQLAlchemy and most libs we use. Add manual spans only for domain logic the framework can't see (process_order, validate_invoice).
2. Three signals, one SDK. Traces, metrics, and logs all go out via OTLP. Don't add structlog or a Prometheus client — the OTel Python SDK's stdlib-logging handler and metrics API cover both.
3. Sample in the Collector, not the SDK. Use the default AlwaysOn sampler in the SDK. Tail/head sampling lives in the Collector where the outcome of a request is known.
4. Cardinality is the cost driver. Metric attribute values must be bounded. http.route not url.path; user.tier not user.id. Per-request identifiers go on spans and logs, not metrics.
5. Semconv names first, custom names last. Search the Attribute Registry before inventing. Custom attributes use a reverse-DNS namespace (com.rask.order.priority).
6. Set status ERROR only on final failure. Retried-then-succeeded operations stay UNSET. Every ERROR carries a status message with the error class.
7. Resource attributes identify the producer. service.name, service.version, deployment.environment.name. Set via env (OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES), never per-span.

Quick start (Python)

uv add opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

export OTEL_SERVICE_NAME="rask-api"
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.2.3,deployment.environment.name=production"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"
export OTEL_METRICS_EXPORTER="otlp"  # traces & logs default to otlp; metrics doesn't
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED="true"

opentelemetry-instrument python -m rask.api

For development without a Collector, swap exporters to the console:

export OTEL_TRACES_EXPORTER=console OTEL_METRICS_EXPORTER=console OTEL_LOGS_EXPORTER=console

Full details in references/python-sdk.md.

Top gotchas

• Sync code inside an async handler blocks the loop. Auto-instrumentation can't fix a requests.get(...) inside an async route — use httpx.AsyncClient. See sibling python-infrastructure.
• Trace context is lost across queues. Publishing to NATS JetStream drops the span unless you propagate.inject(carrier) into message headers and propagate.extract on the consumer. See python-infrastructure/references/observability.md.
• opentelemetry-instrument is required. Just installing the packages does nothing — the SDK isn't activated until the launcher runs. Missing instrumentations? Re-run opentelemetry-bootstrap -a install in the same venv.
• No metrics by default. OTEL_METRICS_EXPORTER defaults to none in Python (unlike traces/logs). Set it explicitly to otlp.
• Span events for exceptions are deprecated. Don't call span.record_exception(e) for new code. Emit a log record with exception.type/exception.message/exception.stacktrace inside the active span context — it'll carry trace_id/span_id automatically.
• opentelemetry-instrument registers an atexit shutdown hook automatically. No atexit.register(provider.shutdown) needed unless you set up the SDK programmatically.

References

• OpenTelemetry Python docs
• Attribute Registry
• Semantic Conventions spec
• OTel Collector docs
• Instrumentation Score spec — vendor-neutral quality score