Auditor

NeMo Auditor CLI Reference

Environment

• API server: http://localhost:8080 (default)
• Default workspace/namespace: default

Audit Config Commands

# List workspace configs
nemo auditor configs list

# List workspace configs
nemo auditor configs list --workspace <ws>

# Create a config with a JSON payload
# The -d body includes description plus plugins, reporting, run, and system sections.
nemo auditor configs create <name> \
  -d '{"description": "<description>", "plugins": {"probe_spec": "dan.AutoDANCached"}, "reporting": {}, "run": {}, "system": {"lite": true}}'

# Get a config
nemo auditor configs get <name>

# Update a config
nemo auditor configs update <name> \
  -d '{"description": "<new description>", "plugins": {"probe_spec": "dan.AutoDANCached"}, "reporting": {}, "run": {}, "system": {"lite": true}}'

# Delete a config
nemo auditor configs delete <name>

Config JSON Structure

Minimal example for each required field:

• plugins: {"probe_spec": "dan.AutoDANCached"} — specifies which probes to run
• reporting: {}
• run: {} or {"generations": 5}
• system: {"lite": true}

Common probe specs: dan.AutoDANCached, dan.DanInTheWild, dan.goodside

Audit Target Commands

# Create a target
nemo auditor targets create <name> \
  -d '{"model": "<model-name>", "type": "<type>", "description": "<description>"}'

# Create a target with a provider (for real inference endpoints)
nemo auditor targets create <name> \
  -d '{"model": "<model-name>", "type": "<type>", "options": {"provider": "<provider-name>"}}'

# List targets
nemo auditor targets list

# Get a target
nemo auditor targets get <name>

# Update a target
nemo auditor targets update <name> \
  -d '{"model": "<model-name>", "type": "<type>", "description": "<new description>"}'

# Delete a target
nemo auditor targets delete <name>

Target types: nim, openai

Audit Job Commands

# Run an audit locally (spec references config and target as namespace/name)
nemo auditor audit run \
  --spec '{"config": "default/<config-name>", "target": "default/<target-name>"}'

# Submit an audit to a configured cluster
nemo auditor audit submit \
  --spec '{"config": "default/<config-name>", "target": "default/<target-name>"}'

Jobs may take a long time or remain in pending/created status. That is expected.

Typical Workflows

Config CRUD

1. nemo auditor configs list — inspect configs in the active workspace
2. nemo auditor configs create my-config -d '{...}' — create
3. nemo auditor configs get my-config — verify
4. nemo auditor configs update my-config -d '{...}' — update
5. nemo auditor configs delete my-config — delete

Run an Audit Job

1. Create a target pointing to the model endpoint
2. Create a config with probe selection
3. Create a job referencing default/<config> and default/<target>
4. Run locally with nemo auditor audit run --spec '{...}', or submit with nemo auditor audit submit --spec '{...}'