Nemo Experiments Upload

Log evaluation data to NeMo Intake

Get evaluation runs into the platform end-to-end: create an Experiment → create an Evaluation → log traces + scores to an ingest endpoint → see the rollups. The API calls the parent (the leaderboard) an Experiment and each row an Evaluation; the whole feature is called Experiments.

Everything below uses ${NMP_BASE_URL} (default http://localhost:8080) and a ${WORKSPACE} (default default). Point NMP_BASE_URL at the local platform or a remote HTTPS origin. Reject non-loopback http:// targets, and never send authentication across an HTTP redirect. All routes are under /apis/intake/v2/workspaces/${WORKSPACE}.

Pre-flight

Confirm the target platform is reachable before doing anything. If this fails, report the target as unreachable and stop; route to setup/nemo-status only for a local platform.

set -euo pipefail
: "${NMP_BASE_URL:=http://localhost:8080}"
: "${WORKSPACE:=default}"
nmp_authority=${NMP_BASE_URL#*://}
nmp_authority=${nmp_authority%%/*}
case "${nmp_authority}" in
  *@*) echo "NMP_BASE_URL must not contain userinfo" >&2; exit 1 ;;
esac
case "${NMP_BASE_URL}" in
  https://*) ;;
  http://*)
    case "${nmp_authority}" in
      localhost|127.0.0.1) ;;
      localhost:*|127.0.0.1:*)
        nmp_port=${nmp_authority#*:}
        case "${nmp_port}" in
          ""|*[!0-9]*) echo "loopback NMP_BASE_URL has an invalid port" >&2; exit 1 ;;
        esac
        ;;
      *) echo "HTTP NMP_BASE_URL must use exactly localhost or 127.0.0.1" >&2; exit 1 ;;
    esac
    ;;
  *) echo "remote NMP_BASE_URL must use https://" >&2; exit 1 ;;
esac
if curl -sf "${NMP_BASE_URL}/health/ready" >/dev/null; then
  echo "platform ready"
else
  echo "NOT READY — target platform is unreachable or not ready" >&2
  exit 1
fi

What you do

Run the steps in order. Steps 1–2 create the entities; step 3 logs the data; steps 4–5 verify.

1. Create an Experiment

An Experiment is the leaderboard container. You need its id for the next step.

set -euo pipefail
: "${NMP_BASE_URL:=http://localhost:8080}"
: "${WORKSPACE:=default}"
experiments="${NMP_BASE_URL}/apis/intake/v2/workspaces/${WORKSPACE}/experiments"
# Create the experiment. 201 = created, 409 = already exists; any other status is a real failure.
code=$(curl -s -o /dev/null -w '%{http_code}' -X POST "${experiments}" \
  -H 'Content-Type: application/json' \
  -d '{"name": "my-experiment", "description": "example run"}')
case "${code}" in
  201|409) ;;
  *) echo "experiment create failed: HTTP ${code}" >&2; exit 1 ;;
esac
# Fetch the id (works whether it was just created or already existed).
EXPERIMENT_ID=$(curl -sf "${experiments}/my-experiment" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')
[ -n "${EXPERIMENT_ID}" ] || { echo "could not resolve experiment id" >&2; exit 1; }
echo "experiment id: ${EXPERIMENT_ID}"

The POST accepts only 201 (created) or 409 (already exists) — any other status stops the step instead of masking it. The id is then read with a GET, so this works on both first run and re-run; set -euo pipefail + the [ -n ] guard keep it from continuing with an empty EXPERIMENT_ID.

2. Create an Evaluation

Create one Evaluation for each agent/config run against a dataset (one leaderboard row).

curl -sf -X POST \
  "${NMP_BASE_URL}/apis/intake/v2/workspaces/${WORKSPACE}/evaluations" \
  -H 'Content-Type: application/json' \
  -d "{\"name\": \"my-eval-baseline\",
       \"experiment_ids\": [\"${EXPERIMENT_ID}\"],
       \"dataset_name\": \"my-dataset\",
       \"dataset_version\": \"v1\",
       \"metadata\": {\"model\": \"provider/model\", \"job_name\": \"baseline\"}}"

• experiment_ids is a list holding the Experiment's id (from step 1).
• metadata values must be strings (dict[str, str]).
• You must create the Evaluation before you can log to it — ingesting with an unknown evaluation_name returns 400 "…must be created before it can be logged."

3. Log traces + evaluator results

Pick the ingest endpoint that matches your producer. Read ../nemo-intake/references/ingest-formats.md for the full schema and a copy-pasteable example for each. How you attach evaluation identity depends on the endpoint:

• ATIF and chat-completions (JSON body) — add an evaluation_context = {evaluation_name: "<evaluation-name>", test_case_name: "<test-case-name>"} object to the payload.
• OTLP — there is no body field; set nemo.evaluation.name and nemo.test_case.name as attributes on the root span. Spans missing these still ingest but won't associate to an Evaluation.

Evaluator scores arrive one of two ways (both covered in the references):

• Automatically with ATIF — put rewards under extra.verifier_result.rewards (one key per criterion).
• Explicitly — POST .../evaluator-results with {span_id, session_id, name, data_type, value} — use string_value instead of value for CATEGORICAL/TEXT results (see ../nemo-intake/references/ingest-formats.md).

4. Verify the data landed

curl -sf "${NMP_BASE_URL}/apis/intake/v2/workspaces/${WORKSPACE}/evaluations/my-eval-baseline" \
  | python3 -m json.tool

Then list the ingested sessions:

curl -sf "${NMP_BASE_URL}/apis/intake/v2/workspaces/${WORKSPACE}/evaluations/my-eval-baseline/sessions" \
  | python3 -m json.tool

5. View in Studio

Open Studio → the Experiments area (behind the VITE_FF_EXPERIMENT flag) → your experiment → your evaluation. You'll see the leaderboard row with score/cost/latency rollups and can drill into individual sessions and traces.

Reference files

Read these before hand-writing a payload:

• ../nemo-intake/references/ingest-formats.md — the shared Intake request schemas, evaluation context, evaluator results, and examples for ATIF, chat-completions, and OTLP.
• references/harbor-quickstart.md — the Harbor path specifically: mapping a Harbor trial result → an ATIF payload, including verifier rewards → evaluator scores.
• references/troubleshooting.md — every common 400/422/503 from the ingest and CRUD endpoints, with the fix.

Verification

You succeeded when GET .../evaluations/my-eval-baseline shows:

• run_count ≥ 1 (each ingested session counts as one run), and
• non-empty evaluator_names / aggregate_scores if you logged rewards, and/or cost_usd if your spans carried cost.

If run_count is 0 after ingesting, the traces didn't associate — almost always a wrong evaluation identity: evaluation_context.evaluation_name for ATIF/chat-completions, or the nemo.evaluation.name root-span attribute for OTLP (see Gotchas).

If verification fails

Gotchas

• Create before you log. The Evaluation entity must exist before any ingest referencing it — otherwise 400.
• The parent lives at /experiments; /experiment-groups is a deprecated hidden alias. Prefer /experiments. Evaluations are created and logged under /evaluations.
• metadata is dict[str, str] — stringify non-string values or you'll get a 422.
• ATIF and chat-completions are extra="forbid" (unknown keys → 422); evaluation_context itself is lenient (extra="ignore").