Instrument a GenAI app for Dynatrace
This is the shared decision process behind the per-framework recipes in
dynatrace-oss/dynatrace-ai-agent-instrumentation-examples
— referred to throughout as the examples repo — factored out so it applies to an
SDK that repo doesn't cover yet. It tells you which recipe shape to reach for and why,
and what "done" looks like regardless of the SDK/language. Once spans are flowing, load
dt-obs-genai to analyze them.
When you should not use this skill
- Already instrumented, and you want to query/debug GenAI telemetry → dt-obs-genai.
- You already know the SDK's recipe and just want the copy-paste code → go straight to the examples repo; this skill decides which recipe shape applies, not replaces it.
- The app isn't Python or TypeScript → out of scope for this skill.
Where this skill must stop and ask
- Any edit to the customer's application code — show the diff, get a yes, first.
- Credentials — the customer creates the Dynatrace token and provider keys. Never print, log, or store a token value; reference it only via env var or secret store.
- Deploying an OpenPipeline pipeline to their tenant.
- Provider config this skill can't infer — Azure deployment name/API version, Ollama model tag, Bedrock region/profile.
- Adding a collector to their runtime — out of scope; infrastructure, not configuration.
- Enabling message-content capture — off by default; only enable it if the user asks.
Prompts/completions can carry PII or sensitive data, and the knob is SDK-specific:
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true for some instrumentors; a
framework-native flag the env var does not control and that can default to on for others
(e.g. pydantic-ai's InstrumentationSettings(include_content=...)); OpenInference captures
content by default; and Traceloop/OpenLLMetry opentelemetry-instrumentation-<sdk> packages read
TRACELOOP_TRACE_CONTENT, default ON, ignoring the OTel-standard var. Check the SDK's actual
toggle so you don't ship capture on unnoticed; agree on redaction if needed.
The four steps
detect ─▶ instrument ─▶ enrich ─▶ validate
│ │
└──────────── feedback loop ───────┘
(validate failure → fix in enrich or instrument)
1. Detect
Figure out what you're instrumenting before touching any code.
Inputs:
- Dependency manifest (
pyproject.toml / requirements.txt / package.json) — which
LLM SDK(s), agent framework, or observability SDK are present (OpenAI, Anthropic,
Bedrock, LangChain/LangGraph, CrewAI, MCP server/client, Langfuse, ...). An app can
depend on more than one — see references/multi-sdk-apps.md.
- Whether the host already runs Dynatrace OneAgent (
oneagent process/package
present) — if so, OneAgent auto-instruments common HTTP/SDK calls with zero code
changes, which changes everything downstream.
- Whether the app already has any OTel wiring — an existing global
TracerProvider,
an existing Traceloop.init()/instrumentor call, or existing custom span code.
Outputs: a routing key (framework/SDK, existing-instrumentation-state) per SDK
(see references/multi-sdk-apps.md if more than one), a
branch decision (below), and a native-OTel yes/no decision (step 2) per SDK.
How to decide native-OTel vs. fallback: prefer SDKs with a pure-OTel path (native
gen_ai.*). An SDK without one falls back to OpenInference — which closes the loop only if it
emits gen_ai.* (Python semconv dual-write, step 2), else it stops at a normalizer gate. Work
through these in order and stop at the first hit:
- Check
references/examples-index.md for the SDK. A listed opentelemetry/ recipe is
the fast path — use it.
- If the SDK isn't listed, check the OpenTelemetry registry
and the vendor SDK's own docs for native OTel/
gen_ai.* support (e.g. an otel
extra, a span_exporter/tracer-provider hook, or built-in semconv attributes).
Finding one → instrument it (step 2).
- No native OTel path → OpenInference. Instrument with the matching
openinference-instrumentation-<sdk> package from an examples-repo recipe; if not even that
exists, hand-write manual gen_ai.* spans
(references/no-native-instrumentation.md).
OpenInference closes the loop only when it emits gen_ai.* — a Python instrumentor with
semconv dual-write on (step 2). Otherwise (TS/node, older version, unset) spans stay
llm.*/openinference.* and the skill stops at a normalizer gate
(references/enrich-normalization.md), then resumes.
2. Instrument
Wire up span emission. Prefer the pure-OTel path; fall back to OpenInference only when
no native OTel path exists. That fallback closes the loop only when it emits gen_ai.*
(semconv dual-write, below); otherwise it stops at a normalizer gate before Validate.
- Pure OTel (preferred, supported end-to-end). A native OTel instrumentation package, or
the framework's own OTel integration, emitting
gen_ai.* directly — exported straight to the
Dynatrace OTLP endpoint (host + scope rules in the common shape below) or via a collector. Many
agent frameworks and vendor SDKs ship one (or accept a custom span_exporter); the current list
lives in the examples repo, not here. ⚠️ Prefer the genuine OTel-published instrumentor over a
Traceloop/OpenLLMetry one — Traceloop/OpenLLMetry is reportedly sunset upstream, so don't use it
for new instrumentation (treat an app already on it as enhance-existing/migration). It also
defaults message-content capture on; which package and how to route it live in
references/examples-index.md §1a.
- Fallback = OpenInference. No native OTel path → the matching
openinference-instrumentation-<sdk> package from an examples-repo recipe
(OpenAIInstrumentor, AnthropicInstrumentor, ...). Before initializing it, explicitly
disable that SDK's default-on message-content capture using its documented setting; do not
rely on the OTel-standard env var. If capture cannot be disabled, stop and ask before
running a request. Python instrumentors can dual-write
it validates like native. Otherwise (TS/node, older version, or unset) spans stay
llm.*/openinference.* and the skill stops before Validate until the user applies a
normalizer (references/enrich-normalization.md). No
OpenInference package either → hand-write manual spans
(references/no-native-instrumentation.md).
- OneAgent present: skip this step — no code changes, OneAgent auto-instruments.
Only give the app/service a unique title (e.g.
FastAPI(title=...)) so
Smartscape creates a distinct entity instead of merging it with another app.
Common shape, in greenfield mode (when not using OneAgent): match the app's existing
package manager; if creating from scratch (Python), default to uv.
- Build a
Resource with at least service.name, and create a TracerProvider with
it. gen_ai.agent.name and other agent-identity attrs are span-level, not
Resource-level — the GenAI views and agent quick filter read them off the span, so
set them at span creation or in a span processor (see Enrich), not on the Resource.
- Attach a span processor + OTLP exporter pointed at either Dynatrace directly or a
local OTel Collector (
http://localhost:4318) when Enrich needs a transform first.
The endpoint form depends on the exporter — they are not interchangeable:
- A collector's
otlphttp exporter takes the base path ($DT_ENDPOINT/api/v2/otlp)
and appends /v1/<signal> itself.
- An SDK-native exporter (e.g. Python's
OTLPSpanExporter(endpoint=...), Node's
OTLPTraceExporter) uses the endpoint verbatim as the POST URL — pass the full
$DT_ENDPOINT/api/v2/otlp/v1/traces (or /v1/metrics, /v1/logs), or it 404s.
Header: Authorization: Api-Token $DT_API_TOKEN, token scope openTelemetryTrace.ingest
plus metrics.ingest/logs.ingest if needed. Never print, log, or store the token value.
- Call the instrumentor / SDK init before the app's first LLM call, not lazily
inside the request handler, and wrap the call so session/conversation correlation
can be attached (see Enrich).
Inputs: routing key + native-OTel decision from step 1; DT_ENDPOINT/DT_API_TOKEN or
collector endpoint; service name.
Outputs: a running app emitting spans — gen_ai.* directly (pure OTel), or
llm.*/openinference.* pending a user-side normalizer (fallback).
3. Enrich
Get spans to full gen_ai.* semantic-convention compliance and add cross-cutting context.
Always do, regardless of path:
- Set metrics export to delta temporality (
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta) — Dynatrace rejects cumulative metrics with a 400.
- Message-content capture (
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY or
SPAN_AND_EVENT, a framework-native flag, or OpenInference's default-on capture) is a
stop-and-ask decision — see above. Once approved, confirm gen_ai.input.messages /
gen_ai.output.messages land.
- Set a stable conversation ID once per conversation, not regenerated per request,
targeting
gen_ai.conversation.id. Prefer a framework-native mechanism where one exists
(e.g. pydantic-ai derives gen_ai.conversation.id from its run/message history through its
instrumentation); else the language-specific span mutation API at the call site if you create
the span yourself (set_attribute in Python or setAttribute in Node — e.g. the
no-native-instrumentation manual-span path); for an auto-instrumenting baseline package that
creates spans internally with no call-site handle, register the language-specific span-processor
hook (on_start in Python, onStart in Node) and set the attribute before the span is exported
— the same mechanism works for any attribute the instrumentor omits that your app can supply.
In multi-agent apps, thread the same id through every agent/sub-agent call — else each run
emits its own id and the conversation fragments into pieces sharing only a trace.id.
OpenInference users: using_attributes(session_id=...) emits session.id, not
gen_ai.conversation.id, so confirm the
normalization rule actually maps it.
- For agent frameworks, expect nested spans (crew/workflow → agent → task/tool);
derive duration metrics from span-name/span-kind, not
gen_ai.operation.name — CrewAI
sets that to invoke_agent on all three levels at once, folding them into one histogram.
Fallback path (OpenInference) — user-side normalization gate: when semconv dual-write
isn't available (TS/node, older versions), spans arrive as llm.*/openinference.*, not
gen_ai.*, so Validate cannot pass until they're normalized — and the skill can't do this
step for the user. Hand them the example normalizer config (collector gen_ai_normalizer or
OpenPipeline — see references/enrich-normalization.md),
and only resume Validate once they confirm it's applied.
Inputs: raw spans from step 2 — gen_ai.* (pure OTel), or llm.*/openinference.*
awaiting the user-side normalizer (fallback).
Outputs: gen_ai.*-compliant spans, with conversation correlation, plus any derived
agent/workflow duration or token-usage metrics.
4. Validate
Confirm the pipeline works end-to-end, against a known contract — don't eyeball it.
dtctl is the primary loop: it queries the emitted spans in Dynatrace so the
assertion is mechanical and repeatable. The full recipe, contract pin, and severity rules
live in references/validate-contract.md.
Inputs: the running, instrumented app from steps 2–3. On the OpenInference fallback,
only start Validate after the user confirms their normalizer is applied (the Enrich
gate) — otherwise every check below fails on llm.* spans that were never mapped.
Steps:
- Send a real request through the app.
- Check
dtctl is installed (dtctl version); if not, install it from the
dtctl README before continuing.
- Query the resulting spans with
dtctl, filtered by service.name — see the ready
dtctl query in references/validate-contract.md.
(Manual fallback if dtctl is unavailable: the same fetch spans DQL in the AI
Observability app's Explorer / Distributed Tracing app — load dt-dql-essentials for
query syntax — but it loses the repeatable assertion dtctl gives you.)
- Assert the mandatory fields and metrics declared by the contract profile matching the app's
provider — at minimum
gen_ai.provider.name/gen_ai.system (any-of; prefer emitting
gen_ai.provider.name). Treat gen_ai.operation.name as recommended unless the selected
profile/scenario marks it required, and apply inference/provider-specific fields only where
that profile says they apply. Severity (fail vs. warn) and per-provider profiles:
references/validate-contract.md.
Outputs: pass/fail against the contract. On failure, the fix belongs in Enrich
(missing/wrong attribute mapping) or Instrument (wrong SDK/collector config) — not
in Validate itself; Validate only ever detects, it doesn't patch. After fixing, re-run the
same dtctl query to confirm.
OneAgent path: steps 1–4 are unchanged — still query with dtctl — but a failure has
no Instrument/Enrich step to route to: it's either an unfixable OneAgent-version gap
(report it) or a non-unique step 2 entity title (fix it). Once Validate passes, load
dt-obs-genai to analyze.
Greenfield vs. enhance-existing
Both branches run the same four steps — they differ in what Detect finds and what
Instrument is allowed to touch. Default to enhance-existing; greenfield is the narrow
exception.
Enhance-existing — the task targets an existing repository, application, template,
example collection, workflow, or configuration, even when the specific file or example you're
asked to add doesn't exist yet. There is a contract to preserve — a sibling implementation, an
instrumentation setup, a build/dependency convention, a test harness, a deployment config — so
conform to it; don't invent a parallel one. The greenfield "Common shape" is not safe to
copy verbatim: a fresh Resource/TracerProvider silently replaces an existing one and can
drop spans the app relies on. See references/enhance-existing.md
for the per-case breakdown and the service-naming collision rule.
Greenfield — applies only when there is no existing repository contract, sibling
implementation, instrumentation setup, build convention, test harness, or deployment
configuration to preserve. Only then does Instrument create a fresh
Resource/TracerProvider/exporter with no risk of clobbering anything. If in doubt, it's
enhance-existing — match what's there rather than scaffold something new.
References
- dynatrace-oss/dynatrace-ai-agent-instrumentation-examples — concrete per-framework recipes this backbone is distilled from
- OTel GenAI semantic conventions — the
gen_ai.* attribute contract this skill instruments toward
- OpenTelemetry registry — used in step 1's native-OTel-vs-fallback decision
- references/examples-index.md — routing key → concrete recipe in the examples repo
- references/enrich-normalization.md — the OpenInference fallback gate: when normalization is needed before Validate can pass (and when the semconv flag skips it), plus the ready-made normalizer config (collector / OpenPipeline) the user applies
- references/multi-sdk-apps.md — running Detect/Instrument/Enrich when an app depends on more than one LLM SDK
- references/no-native-instrumentation.md — manual pure-OTel
gen_ai.* spans as a stopgap when no packaged instrumentation exists for an SDK at all
- references/enhance-existing.md — full reasoning for each enhance-existing case
- references/validate-contract.md — the contract (read on
main), the dtctl validation loop, fail/warn severity, the any-of gate, provider profiles, and derived-metric lag
1---2name: dt-setup-genai3description: >-4 Framework-independent decision process for instrumenting a GenAI/LLM app (Python or5 TypeScript, any provider/agent SDK) so it emits OTel GenAI semantic-convention6 (gen_ai.*) spans queryable in Dynatrace. Four steps: detect, instrument, enrich,7 validate. Decides which approach a given SDK needs; does not replace the concrete8 per-framework recipes in dynatrace-oss/dynatrace-ai-agent-instrumentation-examples.9 Trigger: "instrument this GenAI app", "add tracing to this LLM app", "how do I get10 gen_ai spans into Dynatrace", "onboard this agent app to AI Observability". Do NOT11 use for analyzing telemetry from an already-instrumented app (use dt-obs-genai) or12 for languages other than Python/TypeScript.13license: Apache-2.014metadata:15 source: https://skillmd.com/skills/gabrielmoreira/dt-setup-genai16---1718# Instrument a GenAI app for Dynatrace1920This is the shared decision process behind the per-framework recipes in21[`dynatrace-oss/dynatrace-ai-agent-instrumentation-examples`](https://github.com/dynatrace-oss/dynatrace-ai-agent-instrumentation-examples)22— referred to throughout as **the examples repo** — factored out so it applies to an23SDK that repo doesn't cover yet. It tells you which recipe shape to reach for and why,24and what "done" looks like regardless of the SDK/language. Once spans are flowing, load25**dt-obs-genai** to analyze them.2627## When you should not use this skill2829- Already instrumented, and you want to query/debug GenAI telemetry → **dt-obs-genai**.30- You already know the SDK's recipe and just want the copy-paste code → go straight to the examples repo; this skill decides *which* recipe shape applies, not replaces it.31- The app isn't Python or TypeScript → out of scope for this skill.3233## Where this skill must stop and ask3435- **Any edit to the customer's application code** — show the diff, get a yes, first.36- **Credentials** — the customer creates the Dynatrace token and provider keys. Never print, log, or store a token value; reference it only via env var or secret store.37- **Deploying an OpenPipeline pipeline** to their tenant.38- **Provider config this skill can't infer** — Azure deployment name/API version, Ollama model tag, Bedrock region/profile.39- **Adding a collector to their runtime** — out of scope; infrastructure, not configuration.40- **Enabling message-content capture** — **off by default; only enable it if the user asks.**41 Prompts/completions can carry PII or sensitive data, and the knob is **SDK-specific**:42 `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` for some instrumentors; a43 framework-native flag the env var does **not** control and that can **default to on** for others44 (e.g. pydantic-ai's `InstrumentationSettings(include_content=...)`); OpenInference captures45 content by default; and Traceloop/OpenLLMetry `opentelemetry-instrumentation-<sdk>` packages read46 `TRACELOOP_TRACE_CONTENT`, default **ON**, ignoring the OTel-standard var. Check the SDK's actual47 toggle so you don't ship capture on unnoticed; agree on redaction if needed.4849## The four steps5051```52detect ─▶ instrument ─▶ enrich ─▶ validate53 │ │54 └──────────── feedback loop ───────┘55 (validate failure → fix in enrich or instrument)56```5758### 1. Detect5960Figure out *what* you're instrumenting before touching any code.6162**Inputs:**63- Dependency manifest (`pyproject.toml` / `requirements.txt` / `package.json`) — which64 LLM SDK(s), agent framework, or observability SDK are present (OpenAI, Anthropic,65 Bedrock, LangChain/LangGraph, CrewAI, MCP server/client, Langfuse, ...). An app can66 depend on more than one — see [references/multi-sdk-apps.md](references/multi-sdk-apps.md).67- Whether the host already runs Dynatrace **OneAgent** (`oneagent` process/package68 present) — if so, OneAgent auto-instruments common HTTP/SDK calls with zero code69 changes, which changes everything downstream.70- Whether the app already has *any* OTel wiring — an existing global `TracerProvider`,71 an existing `Traceloop.init()`/instrumentor call, or existing custom span code.7273**Outputs:** a **routing key** `(framework/SDK, existing-instrumentation-state)` per SDK74(see [references/multi-sdk-apps.md](references/multi-sdk-apps.md) if more than one), a75**branch decision** (below), and a **native-OTel yes/no** decision (step 2) per SDK.7677**How to decide native-OTel vs. fallback:** **prefer** SDKs with a pure-OTel path (native78`gen_ai.*`). An SDK without one falls back to OpenInference — which closes the loop only if it79emits `gen_ai.*` (Python semconv dual-write, step 2), else it stops at a normalizer gate. Work80through these in order and stop at the first hit:811. Check `references/examples-index.md` for the SDK. A listed `opentelemetry/` recipe is82 the fast path — use it.832. If the SDK isn't listed, check the [OpenTelemetry registry](https://opentelemetry.io/ecosystem/registry/)84 and the vendor SDK's own docs for native OTel/`gen_ai.*` support (e.g. an `otel`85 extra, a `span_exporter`/tracer-provider hook, or built-in semconv attributes).86 Finding one → instrument it (step 2).873. **No native OTel path → OpenInference.** Instrument with the matching88 `openinference-instrumentation-<sdk>` package from an examples-repo recipe; if not even that89 exists, hand-write manual `gen_ai.*` spans90 ([references/no-native-instrumentation.md](references/no-native-instrumentation.md)).91 OpenInference closes the loop **only when it emits `gen_ai.*`** — a Python instrumentor with92 semconv dual-write on (step 2). Otherwise (TS/node, older version, unset) spans stay93 `llm.*`/`openinference.*` and the skill **stops at a normalizer gate**94 ([references/enrich-normalization.md](references/enrich-normalization.md)), then resumes.9596### 2. Instrument9798Wire up span emission. **Prefer the pure-OTel path**; fall back to OpenInference only when99no native OTel path exists. That fallback closes the loop only when it emits `gen_ai.*`100(semconv dual-write, below); otherwise it stops at a normalizer gate before Validate.101102- **Pure OTel (preferred, supported end-to-end).** A native OTel instrumentation package, or103 the framework's own OTel integration, emitting `gen_ai.*` directly — exported straight to the104 Dynatrace OTLP endpoint (host + scope rules in the common shape below) or via a collector. Many105 agent frameworks and vendor SDKs ship one (or accept a custom `span_exporter`); the current list106 lives in the examples repo, not here. ⚠️ Prefer the genuine OTel-published instrumentor over a107 Traceloop/OpenLLMetry one — Traceloop/OpenLLMetry is reportedly sunset upstream, so **don't use it108 for new instrumentation** (treat an app already on it as enhance-existing/migration). It also109 defaults message-content capture on; which package and how to route it live in110 [references/examples-index.md](references/examples-index.md) §1a.111- **Fallback = OpenInference.** No native OTel path → the matching112 `openinference-instrumentation-<sdk>` package from an examples-repo recipe113 (`OpenAIInstrumentor`, `AnthropicInstrumentor`, ...). **Before initializing it, explicitly114 disable that SDK's default-on message-content capture using its documented setting; do not115 rely on the OTel-standard env var. If capture cannot be disabled, stop and ask before116 running a request.** Python instrumentors can dual-write117 it validates like native. **Otherwise** (TS/node, older version, or unset) spans stay118 `llm.*`/`openinference.*` and the skill **stops before Validate** until the user applies a119 normalizer ([references/enrich-normalization.md](references/enrich-normalization.md)). No120 OpenInference package either → hand-write manual spans121 ([references/no-native-instrumentation.md](references/no-native-instrumentation.md)).122- **OneAgent present:** skip this step — no code changes, OneAgent auto-instruments.123 Only give the app/service a **unique title** (e.g. `FastAPI(title=...)`) so124 Smartscape creates a distinct entity instead of merging it with another app.125126**Common shape, in greenfield mode (when not using OneAgent):** match the app's existing127package manager; if creating from scratch (Python), default to **`uv`**.1281. Build a `Resource` with at least `service.name`, and create a `TracerProvider` with129 it. `gen_ai.agent.name` and other agent-identity attrs are **span-level**, not130 Resource-level — the GenAI views and agent quick filter read them off the span, so131 set them at span creation or in a span processor (see Enrich), not on the `Resource`.1322. Attach a span processor + OTLP exporter pointed at either Dynatrace directly or a133 local OTel Collector (`http://localhost:4318`) when Enrich needs a transform first.134 **The endpoint form depends on the exporter — they are not interchangeable:**135 - A collector's `otlphttp` exporter takes the base path (`$DT_ENDPOINT/api/v2/otlp`)136 and appends `/v1/<signal>` itself.137 - An SDK-native exporter (e.g. Python's `OTLPSpanExporter(endpoint=...)`, Node's138 `OTLPTraceExporter`) uses the endpoint **verbatim** as the POST URL — pass the full139 `$DT_ENDPOINT/api/v2/otlp/v1/traces` (or `/v1/metrics`, `/v1/logs`), or it 404s.140 Header: `Authorization: Api-Token $DT_API_TOKEN`, token scope `openTelemetryTrace.ingest`141 plus `metrics.ingest`/`logs.ingest` if needed. Never print, log, or store the token value.1423. Call the instrumentor / SDK init *before* the app's first LLM call, not lazily143 inside the request handler, and wrap the call so session/conversation correlation144 can be attached (see Enrich).145146**Inputs:** routing key + native-OTel decision from step 1; `DT_ENDPOINT`/`DT_API_TOKEN` or147collector endpoint; service name.148**Outputs:** a running app emitting spans — `gen_ai.*` directly (pure OTel), or149`llm.*`/`openinference.*` pending a user-side normalizer (fallback).150151### 3. Enrich152153Get spans to full `gen_ai.*` semantic-convention compliance and add cross-cutting context.154155**Always do, regardless of path:**156- Set metrics export to **delta temporality** (`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta`) — Dynatrace rejects cumulative metrics with a 400.157- Message-content capture (`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY` or158 `SPAN_AND_EVENT`, a framework-native flag, or OpenInference's default-on capture) is a159 **stop-and-ask** decision — see above. Once approved, confirm `gen_ai.input.messages` /160 `gen_ai.output.messages` land.161- Set a **stable conversation ID once per conversation, not regenerated per request**,162 targeting `gen_ai.conversation.id`. **Prefer a framework-native mechanism** where one exists163 (e.g. pydantic-ai derives `gen_ai.conversation.id` from its run/message history through its164 instrumentation); else the language-specific span mutation API at the call site if you create165 the span yourself (`set_attribute` in Python or `setAttribute` in Node — e.g. the166 no-native-instrumentation manual-span path); for an auto-instrumenting baseline package that167 creates spans internally with no call-site handle, register the language-specific span-processor168 hook (`on_start` in Python, `onStart` in Node) and set the attribute before the span is exported169 — the same mechanism works for any attribute the instrumentor omits that your app can supply.170 In **multi-agent apps, thread the same id through every agent/sub-agent call** — else each run171 emits its own id and the conversation fragments into pieces sharing only a `trace.id`.172 OpenInference users: `using_attributes(session_id=...)` emits `session.id`, not173 `gen_ai.conversation.id`, so confirm the174 [normalization rule](references/enrich-normalization.md) actually maps it.175- For agent frameworks, expect **nested spans** (crew/workflow → agent → task/tool);176 derive duration metrics from span-name/span-kind, not `gen_ai.operation.name` — CrewAI177 sets that to `invoke_agent` on all three levels at once, folding them into one histogram.178179**Fallback path (OpenInference) — user-side normalization gate:** when semconv dual-write180isn't available (TS/node, older versions), spans arrive as `llm.*`/`openinference.*`, not181`gen_ai.*`, so Validate cannot pass until they're normalized — and the skill **can't do this182step for the user**. Hand them the example normalizer config (collector `gen_ai_normalizer` or183OpenPipeline — see [references/enrich-normalization.md](references/enrich-normalization.md)),184and only resume Validate once they confirm it's applied.185186**Inputs:** raw spans from step 2 — `gen_ai.*` (pure OTel), or `llm.*`/`openinference.*`187awaiting the user-side normalizer (fallback).188**Outputs:** `gen_ai.*`-compliant spans, with conversation correlation, plus any derived189agent/workflow duration or token-usage metrics.190191### 4. Validate192193Confirm the pipeline works end-to-end, against a known contract — don't eyeball it.194**`dtctl` is the primary loop:** it queries the emitted spans in Dynatrace so the195assertion is mechanical and repeatable. The full recipe, contract pin, and severity rules196live in [references/validate-contract.md](references/validate-contract.md).197198**Inputs:** the running, instrumented app from steps 2–3. On the OpenInference fallback,199only start Validate **after** the user confirms their normalizer is applied (the Enrich200gate) — otherwise every check below fails on `llm.*` spans that were never mapped.201**Steps:**2021. Send a real request through the app.2032. **Check `dtctl` is installed** (`dtctl version`); if not, install it from the204 [dtctl README](https://github.com/dynatrace-oss/dtctl) before continuing.2053. **Query the resulting spans with `dtctl`**, filtered by `service.name` — see the ready206 `dtctl query` in [references/validate-contract.md](references/validate-contract.md).207 (Manual fallback if `dtctl` is unavailable: the same `fetch spans` DQL in the AI208 Observability app's Explorer / Distributed Tracing app — load **dt-dql-essentials** for209 query syntax — but it loses the repeatable assertion `dtctl` gives you.)2104. Assert the mandatory fields and metrics declared by the contract profile matching the app's211 provider — at minimum `gen_ai.provider.name`/`gen_ai.system` (any-of; prefer emitting212 `gen_ai.provider.name`). Treat `gen_ai.operation.name` as recommended unless the selected213 profile/scenario marks it required, and apply inference/provider-specific fields only where214 that profile says they apply. Severity (fail vs. warn) and per-provider profiles:215 [references/validate-contract.md](references/validate-contract.md).216**Outputs:** pass/fail against the contract. On failure, the fix belongs in **Enrich**217(missing/wrong attribute mapping) or **Instrument** (wrong SDK/collector config) — not218in Validate itself; Validate only ever detects, it doesn't patch. After fixing, re-run the219same `dtctl` query to confirm.220221**OneAgent path:** steps 1–4 are unchanged — still query with `dtctl` — but a failure has222no Instrument/Enrich step to route to: it's either an unfixable OneAgent-version gap223(report it) or a non-unique step 2 entity title (fix it). Once Validate passes, load224**dt-obs-genai** to analyze.225226## Greenfield vs. enhance-existing227228Both branches run the same four steps — they differ in what Detect finds and what229Instrument is allowed to touch. **Default to enhance-existing; greenfield is the narrow230exception.**231232**Enhance-existing** — the task targets an **existing** repository, application, template,233example collection, workflow, or configuration, **even when the specific file or example you're234asked to add doesn't exist yet.** There is a contract to preserve — a sibling implementation, an235instrumentation setup, a build/dependency convention, a test harness, a deployment config — so236**conform to it; don't invent a parallel one.** The greenfield "Common shape" is **not safe to237copy verbatim**: a fresh `Resource`/`TracerProvider` silently replaces an existing one and can238drop spans the app relies on. See [references/enhance-existing.md](references/enhance-existing.md)239for the per-case breakdown and the service-naming collision rule.240241**Greenfield** — applies **only** when there is no existing repository contract, sibling242implementation, instrumentation setup, build convention, test harness, or deployment243configuration to preserve. Only then does Instrument create a fresh244`Resource`/`TracerProvider`/exporter with no risk of clobbering anything. **If in doubt, it's245enhance-existing** — match what's there rather than scaffold something new.246247## References248249- [dynatrace-oss/dynatrace-ai-agent-instrumentation-examples](https://github.com/dynatrace-oss/dynatrace-ai-agent-instrumentation-examples) — concrete per-framework recipes this backbone is distilled from250- [OTel GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) — the `gen_ai.*` attribute contract this skill instruments toward251- [OpenTelemetry registry](https://opentelemetry.io/ecosystem/registry/) — used in step 1's native-OTel-vs-fallback decision252- [references/examples-index.md](references/examples-index.md) — routing key → concrete recipe in the examples repo253- [references/enrich-normalization.md](references/enrich-normalization.md) — the OpenInference fallback gate: when normalization is needed before Validate can pass (and when the semconv flag skips it), plus the ready-made normalizer config (collector / OpenPipeline) the user applies254- [references/multi-sdk-apps.md](references/multi-sdk-apps.md) — running Detect/Instrument/Enrich when an app depends on more than one LLM SDK255- [references/no-native-instrumentation.md](references/no-native-instrumentation.md) — manual pure-OTel `gen_ai.*` spans as a stopgap when no packaged instrumentation exists for an SDK at all256- [references/enhance-existing.md](references/enhance-existing.md) — full reasoning for each enhance-existing case257- [references/validate-contract.md](references/validate-contract.md) — the contract (read on `main`), the `dtctl` validation loop, fail/warn severity, the any-of gate, provider profiles, and derived-metric lag