NeMo Compass
Before running
Compass reads telemetry; it cannot create it. Confirm all three:
- The target agent already has traces in Intake. No traces means no Insights.
- The platform is reachable at
NHX_BASE_URL. - The CLI has default and fast Platform models configured through
nemo setup, or the submission supplies explicit model references. The Platform must have Jobs and anagents.executeruntime available to execute the analysis.
An ETHOS.md file is optional. It gives Compass the agent's intent,
constraints, and success criteria. Code and traces don't contain that context.
Without it, Compass can only judge an agent against itself.
What it produces
Analyze an agent's behavior from its own telemetry and record what recurs as Insights.
An Insight is a persistent, named description of one recurring problem, and it is the unit of work the rest of the optimization loop runs on. Each carries:
title— a sentence naming the failure, such as "Retrieval drops relevant context near the token limit"description— the failure mode, the tool or model call it affects, and the conditions that trigger itevidence— supporting trace IDs with optional URLs and span references, so a developer can audit the reasoning and build regression tests
Compass targets at least three representative traces per Insight and appends evidence to an existing Insight rather than filing a near-duplicate. It judges behavior rather than status or scores, so it finds failures in sessions that reported success and passed their evaluations. Two well-evidenced Insights are worth more than ten vague ones, so a run that files nothing is a valid outcome.
Run it
Replace the quoted placeholders below with the target agent and workspace.
nemo compass analysis-runs create --agent "<agent-name>" --workspace "<workspace>" --wait
Add --ethos ETHOS.md to supply the agent's intended behavior, --since for
an ISO-8601 lower time bound, or --evaluation-id to select an evaluation.
The command submits an AnalysisRun backed by an agents.execute job. Expect
several minutes. --wait exits non-zero unless the job completes successfully.
The CLI supplies the configured default and fast models. Run nemo setup if
they are missing, or pass --default-model and --fast-model explicitly.
Where Insights are stored
Compass creates and updates insights in NeMo Helix after analysis. Stored insights appear in Studio's optimizer view for the workspace.
Verify
Replace the quoted placeholders with the submitted run name and workspace:
nemo compass analysis-runs get "<run-name>" --workspace "<workspace>"
Confirm the job completed and inspect its analysis report. A completed run may
produce no new insights. When it records insights, read them back and check
for a clear title, an actionable description, and non-empty evidence.
Listing all insights for the agent can include earlier runs.
curl --fail-with-body \
"$NHX_BASE_URL/apis/compass/v2/workspaces/<workspace>/insights/<insight-id>"
On an authenticated platform pass the token through curl's config, not argv.
Replace <url> with the insight URL above:
printf 'header = "Authorization: Bearer %s"' "$(nemo auth token)" | curl --fail-with-body -K - "<url>"
When it finds nothing
Besides a real "nothing worth filing", three things produce an empty result.
Scoping: Compass reads only what --agent and --workspace together
select, and agent_name is carried on agent-level spans, not on their model and
tool children. Volume: too few traces looks the same as a healthy agent. And
telemetry that captures only the shape of a run, spans without the inputs and
outputs, leaves nothing to judge however many spans there are.