# Nemo Compass

> Analyze an agent's production traces to find recurring failure patterns and record each as an Insight. Surveys spans, evaluator scores, and user feedback across many sessions, clusters similar failures, then files every finding as a titled Insight carrying the trace IDs that evidence the problem. Answers why an agent keeps failing, where it gets things wrong, and the recurring problems hiding in production traces.

- Skill: `nvidia-nemo/nemo-compass` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx get-skill add nvidia-nemo/nemo-compass`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-nemo/nemo-compass/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: Apache-2.0
- Author: NVIDIA NeMo (https://skillmd.com/u/nvidia-nemo)
- Updated: 2026-11-12
- Page: https://skillmd.com/skills/nvidia-nemo/nemo-compass

---


# NeMo Compass

## Before running

Compass reads telemetry; it cannot create it. Confirm all three:

- The target agent already has traces in Intake. No traces means no Insights.
- The platform is reachable at `NHX_BASE_URL`.
- The CLI has default and fast Platform models configured through `nemo setup`,
  or the submission supplies explicit model references. The Platform must have
  Jobs and an `agents.execute` runtime available to execute the analysis.

An `ETHOS.md` file is optional. It gives Compass the agent's intent,
constraints, and success criteria. Code and traces don't contain that context.
Without it, Compass can only judge an agent against itself.

## What it produces

Analyze an agent's behavior from its own telemetry and record what recurs as
Insights.

An Insight is a persistent, named description of one recurring problem, and it
is the unit of work the rest of the optimization loop runs on. Each carries:

- `title` — a sentence naming the failure, such as "Retrieval drops relevant
  context near the token limit"
- `description` — the failure mode, the tool or model call it affects, and the
  conditions that trigger it
- `evidence` — supporting trace IDs with optional URLs and span references, so a developer can
  audit the reasoning and build regression tests

Compass targets at least three representative traces per Insight and appends
evidence to an existing Insight rather than filing a near-duplicate. It judges
behavior rather than status or scores, so it finds failures in sessions that
reported success and passed their evaluations. Two well-evidenced Insights are
worth more than ten vague ones, so a run that files nothing is a valid outcome.

## Run it

Replace the quoted placeholders below with the target agent and workspace.

```bash
nemo compass analysis-runs create --agent "<agent-name>" --workspace "<workspace>" --wait
```

Add `--ethos ETHOS.md` to supply the agent's intended behavior, `--since` for
an ISO-8601 lower time bound, or `--evaluation-id` to select an evaluation.
The command submits an AnalysisRun backed by an `agents.execute` job. Expect
several minutes. `--wait` exits non-zero unless the job completes successfully.

The CLI supplies the configured default and fast models. Run `nemo setup` if
they are missing, or pass `--default-model` and `--fast-model` explicitly.

## Where Insights are stored

Compass creates and updates insights in NeMo Helix after analysis.
Stored insights appear in Studio's optimizer view for the workspace.

## Verify

Replace the quoted placeholders with the submitted run name and workspace:

```bash
nemo compass analysis-runs get "<run-name>" --workspace "<workspace>"
```

Confirm the job completed and inspect its analysis report. A completed run may
produce no new insights. When it records insights, read them back and check
for a clear title, an actionable description, and non-empty `evidence`.
Listing all insights for the agent can include earlier runs.

```bash
curl --fail-with-body \
  "$NHX_BASE_URL/apis/compass/v2/workspaces/<workspace>/insights/<insight-id>"
```

On an authenticated platform pass the token through curl's config, not argv.
Replace `<url>` with the insight URL above:

```bash
printf 'header = "Authorization: Bearer %s"' "$(nemo auth token)" | curl --fail-with-body -K - "<url>"
```

## When it finds nothing

Besides a real "nothing worth filing", three things produce an empty result.
Scoping: Compass reads only what `--agent` and `--workspace` together
select, and `agent_name` is carried on agent-level spans, not on their model and
tool children. Volume: too few traces looks the same as a healthy agent. And
telemetry that captures only the shape of a run, spans without the inputs and
outputs, leaves nothing to judge however many spans there are.

