Apm Integrations

APM Integrations

dd-trace-js provides automatic tracing for 100+ third-party libraries. Each integration consists of two decoupled layers communicating via Node.js diagnostic channels.

Architecture

┌──────────────────────────┐     diagnostic channels      ┌─────────────────────────┐
│     Instrumentation      │ ──────────────────────────▶  │        Plugin           │
│ datadog-instrumentations │    apm:<name>:<op>:start     │  datadog-plugin-<name>  │
│                          │    apm:<name>:<op>:finish    │                         │
│ Hooks into library       │    apm:<name>:<op>:error     │ Creates spans, sets     │
│ methods, emits events    │                              │ tags, handles errors    │
└──────────────────────────┘                              └─────────────────────────┘

finish above is the legacy manual-channel completion event. tracingChannel and Orchestrion use end / asyncEnd, as described below.

Instrumentation (packages/datadog-instrumentations/src/): Hooks into a library's internals and publishes events with context data to named diagnostic channels. Has zero knowledge of tracing — only emits events.

Plugin (packages/datadog-plugin-<name>/src/): Subscribes to diagnostic channel events and creates APM spans with service name, resource, tags, and error metadata. Extends a base class providing lifecycle management.

Both layers are always needed for a new integration.

Instrumentation: Orchestrion First

Orchestrion is the required default when the work exists as a source function. It rewrites matched CJS/ESM source from JavaScript config, avoiding runtime monkey-patching and ESM's static-binding traps. Start there for top-level declarations, class/object methods, named expressions, and assignments to named receivers. Use shimmer only when the work is created entirely at runtime or the required argument/result mutation cannot happen from Orchestrion's subscriber lifecycle.

Config lives in packages/datadog-instrumentations/src/helpers/rewriter/instrumentations/<name>.js. See Orchestrion Reference for the full config format and examples.

When Shimmer Is Necessary Instead

Shimmer (addHook + shimmer.wrap) should only be used when orchestrion cannot handle the pattern. When using shimmer, always include a code comment explaining why orchestrion is not viable. Valid reasons:

• Dynamic method interception — methods created at runtime or on prototype chains that orchestrion's static analysis cannot reach
• Factory results that cannot be substituted — end can replace synchronous results and asyncEnd can replace native-Promise results; shimmer remains necessary for Promise subclasses, userland thenables, or APIs that require the original result's identity
• Pre-lifecycle argument modification — arguments must be changed before Orchestrion's bindStart / subscribers can run

If none of these apply, use orchestrion. For shimmer patterns, refer to existing shimmer-based instrumentations in the codebase (e.g., packages/datadog-instrumentations/src/pg.js). Always try to use Orchestrion when beginning a new integration!

Plugin Base Classes

Plugins extend a base class matching the library type. The base class provides automatic channel subscriptions, span lifecycle, and type-specific tags.

Plugin
├── CompositePlugin              — Multiple sub-plugins (produce + consume)
├── LogPlugin                    — Log correlation injection (no spans)
├── WebPlugin                    — Base web plugin
│   └── RouterPlugin             — Web frameworks with middleware
└── TracingPlugin                — Base for all span-creating plugins
    ├── InboundPlugin            — Inbound calls
    │   ├── ServerPlugin         — HTTP servers
    │   └── ConsumerPlugin       — Message consumers (DSM)
    └── OutboundPlugin           — Outbound calls
        ├── ProducerPlugin       — Message producers (DSM)
        └── ClientPlugin         — HTTP/RPC clients
            └── StoragePlugin    — Storage systems
                ├── DatabasePlugin   — Database clients (DBM, db.* tags)
                └── CachePlugin      — Key-value caches

Wrong base class = complex workarounds. Always match the library type to the base class.

Read Upstream Source First

Touching packages/datadog-instrumentations/src/<lib>.js, its plugin counterpart, or any orchestrion config — for any reason — read the upstream library's source first. Memory of an SDK's contract drifts faster than the SDK; comments in the wrap go stale every minor version; cross-version diffs surface contract changes guessing misses (lazy → eager attachment, mode-exclusive APIs, new error paths).

Two ways to fetch the source locally:

1. Shallow clone the installed version:

git clone --depth 1 --branch v<x.y.z> https://github.com/<org>/<repo>.git /tmp/<lib>-versions/v<x.y.z>

1. npm pack when the published runtime artifact is what matters:

cd /tmp/<lib>-versions && npm pack <lib>@<x.y.z>
tar -xzf <lib>-<x.y.z>.tgz -C v<x.y.z> --strip-components=1

Read the file the wrap hooks, the base classes the hooked methods inherit from, and files the wrap doesn't currently touch — a public method, an internal channel, or a metadata field the current instrumentation skipped often gives a cleaner hook (e.g., kafka cluster.brokerPool.metadata.clusterId, couchbase tracingChannel).

Key Concepts

The ctx Object

Context flows from instrumentation to plugin:

• Orchestrion: automatically provides ctx.arguments (method args) and ctx.self (instance)
• Shimmer: instrumentation sets named properties (ctx.sql, ctx.client, etc.)
• Plugin sets: ctx.currentStore (span), ctx.parentStore (parent span)
• On completion: ctx.result or ctx.error

Channel Event Lifecycle

• runStores() for start events — establishes async context (always)
• publish() for completion/error events — notification only
• hasSubscribers guard — skip publish/subscriber work when no plugin listens; orchestrion still pays wrapper setup in current templates
• When shimmer is necessary, prefer tracingChannel (from dc-polyfill) over manual channels — it provides start/end/asyncStart/asyncEnd/error events automatically

Channel Prefix Patterns

• Orchestrion: tracing:orchestrion:<npm-package>:<channelName> (set via static prefix)
• Shimmer + tracingChannel (preferred): tracing:apm:<name>:<operation> (set via static prefix)
• Shimmer + manual channels (legacy): apm:{id}:{operation} (default, no static prefix needed)

bindStart and completion handlers

Use bindStart to create the span and return its store. Finish in the event the instrumentation emits: usually end for synchronous work, asyncEnd for promises/callbacks, and finish only for legacy instrumentations that publish it. Orchestrion does not publish finish.

Subscriber Cardinality (channel.publish position)

When relocating a channel.publish call behind a dedupe gate, depth filter, cache-hit return, or any short-circuit, the question is not "is the publish still there?" but "what cardinality does each downstream subscriber need?". Subscribers split into two camps that look identical from inside the publish site:

• Once per first occurrence — tracing plugins that dedupe spans, distinct-path metrics. Safe behind a dedupe gate.
• Once per call — IAST taint-tracking (mutates each call's args object by reference), AppSec WAF subscribers that block/log per invocation, anything walking payload identity. Drops data silently when cardinality falls below one-per-call.

Before adding or moving a gate in front of a publish, grep the repo for the channel name, list its subscribers, decide per-subscriber whether the new position preserves the cardinality each needs. When cardinalities diverge, split the publish into a pre-gate (per-call) and a post-gate (per-first-occurrence) call.

Reference Integrations

Always read 1-2 references of the same type before writing or modifying code.

For the complete list by base class, see Reference Plugins.

Debugging

• DD_TRACE_DEBUG=true to see channel activity
• Log Object.keys(ctx) in bindStart to inspect available context
• Spans missing → verify hasSubscribers guard; check channel names match between layers
• Context lost → ensure runStores() (not publish()) for start events
• ESM fails but CJS works → check esmFirst: true in hooks.js (or switch to orchestrion)

Implementation Workflow

Follow these steps when creating or modifying an integration:

1. Investigate — Read the upstream library's source (see Read Upstream Source First). Read 1-2 reference integrations of the same type (see table above). Understand the instrumentation and plugin patterns before writing code.

2. Implement instrumentation — Create the instrumentation in packages/datadog-instrumentations/src/. Use orchestrion for instrumentation.

3. Implement plugin — Create the plugin in packages/datadog-plugin-<name>/src/. Extend the correct base class.

4. Register — Add entries in packages/dd-trace/src/plugins/index.js, every supported public TypeScript surface, docs/test.ts, docs/API.md, and .github/workflows/apm-integrations.yml.

5. Write tests — Add unit tests and ESM integration tests. See Testing for templates.

6. Run tests — Validate with:

   # Run plugin tests (preferred CI command — handles yarn services automatically)
   PLUGINS="<name>" npm run test:plugins:ci
   
   # If the plugin needs external services (databases, message brokers, etc.),
   # check docker-compose.yml for available service names, then:
   docker compose up -d <service>
   SERVICES="<service>" PLUGINS="<name>" npm run test:plugins:ci
   

7. Verify — Confirm all tests pass before marking work as complete.

Reference Files

• New Integration Guide — Step-by-step guide and checklist for creating a new integration end-to-end
• Orchestrion Reference — JavaScript config format, channel naming, function kinds, plugin subscription
• Plugin Patterns — startSpan() API, ctx object details, CompositePlugin, channel subscriptions, code style
• Testing — Unit test and ESM integration test templates
• Reference Plugins — All plugins organized by base class