Llmobs Integrations

dd-trace-py LLMObs Integrations

LLMObs integrations enable Datadog LLM Observability for AI/LLM libraries. They extract model inputs, outputs, token usage, and tool calls from traced spans. This skill should be used in addition to the apm-integrations skill.

Two-Layer Architecture

LLMObs integrations consist of two cooperating layers:

1. Patch Layer (ddtrace/contrib/internal/{name}/patch.py) -- wraps library functions. Standard request/response LLM integrations construct LlmRequestEvent and use core.context_with_event() so the LLM tracing subscriber owns span lifecycle and LLMObs tag extraction.
2. Integration Layer (ddtrace/llmobs/_integrations/{name}.py) -- extends BaseLLMIntegration, implements _set_base_span_tags() and _llmobs_set_tags() to extract and set provider-specific messages, tools, metadata, and token metrics.

Both layers must work together. The patch layer identifies the operation and passes request/response data through the event; the integration layer controls what data is extracted.

Active Patch Patterns

• Event-based request spans: Use LlmRequestEvent with core.context_with_event() for new standard request/response LLM integrations. Anthropic is the canonical reference. This is the preferred pattern.
• Direct integration spans: Some existing or specialized integrations still call integration.trace() and integration.llmobs_set_tags() directly, especially for child spans, agent/tool spans, or integrations not yet migrated. Google GenAI, OpenAI tool spans, and Claude Agent SDK are useful references.

Key Files

Reference Integrations

Always read 1-2 references before writing or modifying LLMObs code.

Use Anthropic as the canonical reference for standard LLM integrations. Use Claude Agent SDK for agent-pattern integrations (agent spans, tool child spans, thinking blocks).

Abstract Methods to Implement

Subclass BaseLLMIntegration and implement:

_set_base_span_tags(span, **kwargs)

Set provider-specific APM tags on the span (e.g., {name}.request.model).

_llmobs_set_tags(span, args, kwargs, response, operation)

Extract and annotate all LLMObs fields on the span:

Fields are usually set via _annotate_llmobs_span_data(...), not raw span._set_ctx_items(...).

Response Metadata

metadata is not request-only. Scalars the provider reports on the response belong there too, so long as they are not already covered by metrics (token counts) or output_messages.

The stop reason is the established case. Record it under the key finish_reason for every provider, so one facet answers "why did generation stop" regardless of integration, and keep each provider's own vocabulary as the value (stop/length/content_filter/tool_calls for the OpenAI family, end_turn/max_tokens/refusal/tool_use for Anthropic). Read it from:

Rules:

• Merge, do not replace. Build the request metadata first, then update it with the response-derived keys (parameters.update(...)), so response data never clobbers request params.
• Omit the key when absent. If the provider reports no reason, leave finish_reason out entirely rather than writing None or "".
• Keep the value a single string. When a request returns multiple choices (n > 1), comma-join the per-choice reasons in choice order ("stop,length") instead of emitting a list, so the key never changes type. _openai_finish_reason_metadata() in _integrations/utils.py does this.
• Guard on the response, not span.error. A response can exist on an errored span (see the AI Guard case in _integrations/anthropic.py); gate on response is not None.

Note two already-shipped integrations predate this key: bedrock and the claude-agent-sdk both write metadata["stop_reason"]. Renaming those is a breaking change and has not been done — follow finish_reason for new work.

Key Constraints

• submit_to_llmobs=True must be set on LlmRequestEvent for event-based request spans or passed to integration.trace() for direct LLMObs spans
• ctx.dispatch_ended_event() must run on success and error paths for event-based patch wrappers
• Streaming must use BaseStreamHandler/AsyncStreamHandler -- never consume streams directly
• Event-based patch wrappers should not call span.set_exc_info(), span.finish(), or integration.llmobs_set_tags() directly; the tracing subscriber handles that when the event ends
• Direct integration spans must keep integration.llmobs_set_tags() and span lifecycle handling aligned with the closest current reference
• Integration instance must be stored on the module: module._datadog_integration = MyLibIntegration(integration_config=config.mylib)

Message Types

from ddtrace.llmobs.types import AudioPart, ImagePart, Message, ToolCall, ToolResult, ToolDefinition

# Input/output messages
Message(content="text", role="user")
Message(content="response", role="assistant", tool_calls=[...])

# Audio attachments in multimodal messages
AudioPart(mime_type="audio/wav", content="<base64-audio>")
Message(content="", role="user", audio_parts=[...])

# Image attachments in multimodal messages
ImagePart(mime_type="image/png", content="<base64-image>")
Message(content="", role="user", image_parts=[...])

# Tool calls (in output messages)
ToolCall(name="get_weather", arguments={"city": "NYC"}, tool_id="toolu_123", type="tool")

# Tool results (in input messages)
ToolResult(result="72F sunny", tool_id="toolu_123", type="tool_result")

# Tool definitions (from request parameters)
ToolDefinition(name="get_weather", description="...", schema={...})

Debugging Quick Tips

• No LLMObs spans -- check submit_to_llmobs=True, ctx.dispatch_ended_event(), and llmobs_enabled
• Wrong messages -- check message extraction handles multi-part content and tool blocks
• Wrong tokens -- check field name mapping (libraries use different names for token counts)
• Streaming broken -- verify BaseStreamHandler subclass, check finalize_stream() dispatches the ended event or finishes direct-trace spans according to the reference pattern
• DD_TRACE_DEBUG=true to see patching activity and span creation

See Failure Modes for detailed debugging guide.

Reference Files

• Implementation Guide -- LLM-specific steps (references apm-integrations guide for the full workflow)
• Failure Modes -- All 8 failure modes with causes and fixes
• Testing Guide -- LLMObs test patterns, VCR cassettes, suitespec