Nemo Intake

Use NeMo Intake

Use Intake to normalize agent telemetry into queryable spans and traces. The Intake endpoint may be local or remote. An agent does not need to run on NeMo Platform; it only needs network access and credentials, when required, to send a supported format to that endpoint.

Requirements

Set the target to the local or remote NeMo Platform origin:

export NMP_BASE_URL=http://127.0.0.1:8080
export WORKSPACE=default

nmp_authority=${NMP_BASE_URL#*://}
nmp_authority=${nmp_authority%%/*}
case "${nmp_authority}" in
  *@*) echo "NMP_BASE_URL must not contain userinfo" >&2; exit 1 ;;
esac
case "${NMP_BASE_URL}" in
  https://*) ;;
  http://*)
    case "${nmp_authority}" in
      localhost|127.0.0.1) ;;
      localhost:*|127.0.0.1:*)
        nmp_port=${nmp_authority#*:}
        case "${nmp_port}" in
          ""|*[!0-9]*) echo "loopback NMP_BASE_URL has an invalid port" >&2; exit 1 ;;
        esac
        ;;
      *) echo "HTTP NMP_BASE_URL must use exactly localhost or 127.0.0.1" >&2; exit 1 ;;
    esac
    ;;
  *) echo "remote NMP_BASE_URL must use https://" >&2; exit 1 ;;
esac

Require:

• Reachable ClickHouse storage.
• A reachable local or remote NeMo Platform intake service with its auth and entities dependencies. ClickHouse must be reachable from Intake; it does not need to be reachable from the telemetry producer.
• One supported telemetry source. NeMo Studio is optional.

For a remote deployment, set NMP_BASE_URL to its HTTPS origin and skip local startup. Use the deployment's authentication mechanism. Do not send authentication across an HTTP redirect: validate the final HTTPS origin and do not add curl -L to authenticated requests. For a local source checkout, follow SETUP.md, make sure Docker is running, then start the backend. Intake automatically provisions local ClickHouse unless NMP_INTAKE_CLICKHOUSE_URL is set:

uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080

Verify the Intake read path before ingesting:

curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1"

Continue only on 200; an empty list is healthy. 503 means Intake cannot reach ClickHouse. For other failures, report the response and route platform startup problems to setup or nemo-status.

Choose an ingest path

All endpoint prefixes are: $NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE.

Read references/ingest-formats.md before hand-writing an ATIF or chat-completions payload, or when you need the semantic attributes and response behavior for OTLP.

Do not translate a native source format without need. Preserve the source's IDs, timestamps, hierarchy, inputs, outputs, statuses, errors, and semantic attributes.

Import an existing trace store

When the user names MLflow, LangSmith, Arize Phoenix, or Braintrust, read only that provider's reference below and run its bundled script. Do not load the other provider references.

Every live import requires an explicit --project, --since, and --until. The scripts write spans first, then provider evaluations and human annotations, verify the imported span IDs, and are safe to replay. They use the existing SDK client factory, including the active CLI context and OAuth token refresh; explicit --nmp-base-url, --workspace, and NMP_ACCESS_TOKEN overrides still work. Use --dry-run to inspect the direct JSON projection without writing Intake.

Direct imports preserve provider timestamps and reject a complete span batch with 422 when any started_at falls outside Intake's 90-day ClickHouse TTL. Increase the spans and trace_index table TTLs before importing older history.

Instrument code for OTLP

Only choose OTLP when the instrumentation emits OpenInference or OTel GenAI semantic conventions. Generic OpenTelemetry spans may ingest, but they do not provide the semantic model, tool, token, cost, input/output, and session fields needed for useful Intake telemetry.

1. Recommend NeMo Relay when it supports the agent runtime and emits a supported semantic format; it avoids hand-written span capture and records agent, model, and tool activity consistently. Use the format Relay actually exports: send semantic OTLP to the OTLP endpoint, or Relay ATIF output to the ATIF endpoint.

2. Otherwise use the runtime's OpenInference instrumentor, or its native OTel GenAI instrumentation, with an OpenTelemetry SDK OTLP exporter. Choose the framework-specific instrumentor from its current documentation; do not invent package names or APIs.

3. If neither semantic convention is available, capture each OpenAI-compatible request and response and send it directly to .../ingest/chat-completions. Do not recommend generic OTLP as a substitute.

4. For supported OTLP instrumentation, point the exporter at Intake:

   export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/otlp/v1/traces"
   export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   

5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. Preserve parent-child IDs and set a stable session.id for related traces. Always set gen_ai.agent.name (or llm.agent.name / agent.name for instrumentation that emits those conventions) on every span to a stable name for the agent. This ensures the spans can be analyzed by NeMo Insights in the future.

6. Run one representative interaction, then verify it through the spans query below.

Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, status, token, cost, and error fields while retaining unhandled attributes.

Customer-facing data model

The hierarchy is session -> trace -> span; evaluator results attach to a span and session.

For ATIF, top-level extra.verifier_result.rewards = {criterion: score} automatically creates a harbor.verifier evaluator span and one evaluator result per criterion. Stock Harbor keeps rewards in a separate reward.json; enrich the trajectory before ingesting or post results explicitly:

curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \
  -H 'Content-Type: application/json' \
  -d '{"span_id":"<span-id>","session_id":"<session-id>","name":"faithfulness/v1","data_type":"NUMERIC","value":0.82}'

Verify ingestion

Always query the interaction back; a successful POST alone is insufficient:

curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=<session-id>&page=1&page_size=100"

Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, status/errors, and any evaluator results. Also query with filter[agent_name]=<agent-name> and confirm the returned agent_name matches the stable value emitted by the agent. If the goal is to create named evaluation runs and compare them in a leaderboard, hand off to nemo-experiments-upload.