Configure Observability Plugins
Start with one exporter managed by the built-in Observability plugin. This is
the default for reusable process configuration and the best first plugin for
most users because it makes Relay's captured activity visible.
Choose one proof output before layering additional telemetry destinations.
Use manual subscriber or exporter APIs only when a test, script, or application
needs direct control over registration names, collection windows, or flush
timing. Both paths consume the same canonical event stream.
Select The Relay Version
Determine whether the application uses NeMo Relay 0.6 or 0.7 before proposing
configuration or binding APIs. Prefer the installed package version, lockfile,
or manifest. Ask the user when the version cannot be established; do not mix
the two surfaces in one example.
- 0.6 uses observability configuration version 2. OpenTelemetry and
OpenInference are separate exporters with
OpenTelemetryConfig /
OpenTelemetrySubscriber and OpenInferenceConfig /
OpenInferenceSubscriber.
- 0.7 uses observability configuration version 3. One typed OpenTelemetry
exporter provides
full, gen_ai, and openinference projections.
Choose The Output
Select the output that best matches the user's immediate inspection target:
- Console or custom event handling
Use a manual subscriber for short-lived in-process inspection.
- Raw canonical lifecycle events
Use ATOF JSONL; read
references/atof.md.
- Portable execution trajectories
Use ATIF; read
references/atif.md.
- OTLP tracing
For 0.6, choose the separate OpenTelemetry or OpenInference exporter. For
0.7, choose a typed OpenTelemetry endpoint (
full, gen_ai, or
openinference). Read references/opentelemetry.md and, for an
OpenInference-aware backend, references/openinference.md.
Choose one output first and verify it before adding another. ATOF is the
default local proof because it preserves the raw event stream with the least
translation. Use synthetic, non-sensitive payloads for the first proof. Add and
verify sanitization before exporters receive production payloads, and never
display complete event records while validating an exporter.
Embedded Event And Subscriber Model
Use this model when explaining how capture and export relate:
- NeMo Relay emits one canonical event stream from scopes, marks, managed tool
calls, managed LLM calls, middleware, and manual lifecycle APIs.
- Subscribers consume events without defining the event model. Multiple
subscribers can observe the same stream for logging, export, analytics, or
diagnostics.
- Global subscribers remain active process-wide until removed.
- Scope-local subscribers are owned by one active scope and disappear when that
scope closes.
- Plugin-installed subscribers are reusable, configuration-driven runtime
components.
- Exporter-oriented subscribers preserve raw ATOF or translate the event stream
into ATIF or the version-appropriate OpenTelemetry/OpenInference form.
- Event payloads reflect sanitized post-guardrail input and output when calls
use managed helpers or manual lifecycle params provide those fields.
- LLM annotations follow the freshness rules:
- Each owning agent scope starts fresh, and a
compaction mark refreshes it.
- The first subsequent LLM start retains complete annotation history. Later
starts retain system instructions, the latest user message, and every
following assistant or tool message.
- When a request codec supplies an annotation, Relay applies the same
event-only projection to provider-shaped event input without changing
provider execution.
- Event fields include semantic input/output through the ATOF
data field,
typed profile data such as model_name and tool_call_id, and codec-provided
annotated LLM request/response data for in-process subscribers and exporters.
- First-class skill tools and the requests to read a complete
SKILL.md
automatically emit skill.load marks under the tool span. The payload
contains only skill_name; metadata records the load source and tool name.
Partial reads do not count, and ambiguous slash-command expansions use the
separate skill.load.inferred name. The eager mark remains present if tool
execution later fails.
Shared Lifecycle
- Create the exporter or subscriber.
- Register it with a unique name before the relevant scoped work.
- Run NeMo Relay-instrumented work inside scopes.
- Flush and deregister in the exporter-specific reference's documented order.
- Shut it down when the process or subsystem is done.
Binding Names
Use the names exported by the selected language binding and Relay version:
- Python 0.6:
nemo_relay.subscribers.register(...), AtofExporter,
AtifExporter, OpenTelemetrySubscriber, and OpenInferenceSubscriber
- Python 0.7: the same registration and file exporters, plus
OpenTelemetrySubscriber for all three typed projections
- Node.js follows the same 0.6/0.7 split through its root exports
- Rust:
nemo_relay::api::subscriber and nemo_relay::observability::*
- Go: source-first wrappers expose equivalent register, exporter, and subscriber
lifecycle methods
Load A Reference When
Load only the reference required by the selected output:
- Load
references/atof.md for raw JSONL events used in local debugging or
offline inspection.
- Load
references/atif.md for ATIF trajectories.
- Load
references/opentelemetry.md for OTLP/OpenTelemetry traces.
- Load
references/openinference.md for the standalone 0.6 OpenInference
exporter or the 0.7 openinference OpenTelemetry projection.
Use Another Skill When
Choose another skill when the task belongs to an adjacent workflow:
- Use
nemo-relay-plugin-build to package subscriber-based export behavior as
a reusable plugin.
- Use
nemo-relay-get-started or nemo-relay-instrument-calls when no scope,
tool call, or LLM call has been instrumented.
- Use
nemo-relay-debug-runtime-integration to diagnose missing telemetry.
Related Skills
Use these skills for adjacent workflows:
- Instrument application calls with
nemo-relay-instrument-calls.
- Add typed wrappers with
nemo-relay-instrument-typed-wrappers.
- Package reusable behavior with
nemo-relay-plugin-build.
- Diagnose missing events with
nemo-relay-debug-runtime-integration.
1---2name: nemo-relay-plugin-observability3description: Use this skill when choosing or configuring NeMo Relay 0.6 or 0.7 observability through the built-in plugin, subscribers, or exporters, including raw ATOF events, ATIF trajectories, OpenTelemetry, OpenInference, or custom event handling.4license: Apache-2.05metadata:6 source: https://skillmd.com/skills/nvidia/nemo-relay-plugin-observability7 author: NVIDIA Corporation and Affiliates8---910# Configure Observability Plugins1112Start with one exporter managed by the built-in Observability plugin. This is13the default for reusable process configuration and the best first plugin for14most users because it makes Relay's captured activity visible.15Choose one proof output before layering additional telemetry destinations.1617Use manual subscriber or exporter APIs only when a test, script, or application18needs direct control over registration names, collection windows, or flush19timing. Both paths consume the same canonical event stream.2021## Select The Relay Version2223Determine whether the application uses NeMo Relay 0.6 or 0.7 before proposing24configuration or binding APIs. Prefer the installed package version, lockfile,25or manifest. Ask the user when the version cannot be established; do not mix26the two surfaces in one example.2728- **0.6** uses observability configuration version 2. OpenTelemetry and29 OpenInference are separate exporters with `OpenTelemetryConfig` /30 `OpenTelemetrySubscriber` and `OpenInferenceConfig` /31 `OpenInferenceSubscriber`.32- **0.7** uses observability configuration version 3. One typed OpenTelemetry33 exporter provides `full`, `gen_ai`, and `openinference` projections.3435## Choose The Output3637Select the output that best matches the user's immediate inspection target:3839- **Console or custom event handling**40 Use a manual subscriber for short-lived in-process inspection.41- **Raw canonical lifecycle events**42 Use ATOF JSONL; read `references/atof.md`.43- **Portable execution trajectories**44 Use ATIF; read `references/atif.md`.45- **OTLP tracing**46 For 0.6, choose the separate OpenTelemetry or OpenInference exporter. For47 0.7, choose a typed OpenTelemetry endpoint (`full`, `gen_ai`, or48 `openinference`). Read `references/opentelemetry.md` and, for an49 OpenInference-aware backend, `references/openinference.md`.5051Choose one output first and verify it before adding another. ATOF is the52default local proof because it preserves the raw event stream with the least53translation. Use synthetic, non-sensitive payloads for the first proof. Add and54verify sanitization before exporters receive production payloads, and never55display complete event records while validating an exporter.5657## Embedded Event And Subscriber Model5859Use this model when explaining how capture and export relate:6061- NeMo Relay emits one canonical event stream from scopes, marks, managed tool62 calls, managed LLM calls, middleware, and manual lifecycle APIs.63- Subscribers consume events without defining the event model. Multiple64 subscribers can observe the same stream for logging, export, analytics, or65 diagnostics.66- Global subscribers remain active process-wide until removed.67- Scope-local subscribers are owned by one active scope and disappear when that68 scope closes.69- Plugin-installed subscribers are reusable, configuration-driven runtime70 components.71- Exporter-oriented subscribers preserve raw ATOF or translate the event stream72 into ATIF or the version-appropriate OpenTelemetry/OpenInference form.73- Event payloads reflect sanitized post-guardrail input and output when calls74 use managed helpers or manual lifecycle params provide those fields.75- LLM annotations follow the freshness rules:76 - Each owning agent scope starts fresh, and a `compaction` mark refreshes it.77 - The first subsequent LLM start retains complete annotation history. Later78 starts retain system instructions, the latest user message, and every79 following assistant or tool message.80 - When a request codec supplies an annotation, Relay applies the same81 event-only projection to provider-shaped event input without changing82 provider execution.83- Event fields include semantic input/output through the ATOF `data` field,84 typed profile data such as `model_name` and `tool_call_id`, and codec-provided85 annotated LLM request/response data for in-process subscribers and exporters.86- First-class skill tools and the requests to read a complete `SKILL.md`87 automatically emit `skill.load` marks under the tool span. The payload88 contains only `skill_name`; metadata records the load source and tool name.89 Partial reads do not count, and ambiguous slash-command expansions use the90 separate `skill.load.inferred` name. The eager mark remains present if tool91 execution later fails.9293## Shared Lifecycle94951. Create the exporter or subscriber.962. Register it with a unique name before the relevant scoped work.973. Run NeMo Relay-instrumented work inside scopes.984. Flush and deregister in the exporter-specific reference's documented order.995. Shut it down when the process or subsystem is done.100101## Binding Names102103Use the names exported by the selected language binding and Relay version:104105- Python 0.6: `nemo_relay.subscribers.register(...)`, `AtofExporter`,106 `AtifExporter`, `OpenTelemetrySubscriber`, and `OpenInferenceSubscriber`107- Python 0.7: the same registration and file exporters, plus108 `OpenTelemetrySubscriber` for all three typed projections109- Node.js follows the same 0.6/0.7 split through its root exports110- Rust: `nemo_relay::api::subscriber` and `nemo_relay::observability::*`111- Go: source-first wrappers expose equivalent register, exporter, and subscriber112 lifecycle methods113114## Load A Reference When115116Load only the reference required by the selected output:117118- Load `references/atof.md` for raw JSONL events used in local debugging or119 offline inspection.120- Load `references/atif.md` for ATIF trajectories.121- Load `references/opentelemetry.md` for OTLP/OpenTelemetry traces.122- Load `references/openinference.md` for the standalone 0.6 OpenInference123 exporter or the 0.7 `openinference` OpenTelemetry projection.124125## Use Another Skill When126127Choose another skill when the task belongs to an adjacent workflow:128129- Use `nemo-relay-plugin-build` to package subscriber-based export behavior as130 a reusable plugin.131- Use `nemo-relay-get-started` or `nemo-relay-instrument-calls` when no scope,132 tool call, or LLM call has been instrumented.133- Use `nemo-relay-debug-runtime-integration` to diagnose missing telemetry.134135## Related Skills136137Use these skills for adjacent workflows:138139- Instrument application calls with `nemo-relay-instrument-calls`.140- Add typed wrappers with `nemo-relay-instrument-typed-wrappers`.141- Package reusable behavior with `nemo-relay-plugin-build`.142- Diagnose missing events with `nemo-relay-debug-runtime-integration`.