Nemo Experimentalist

NeMo Experimentalist

Use the Experimentalist to improve an existing agent. It evaluates a baseline, proposes code changes, and keeps only candidates that improve validation.

Choose the optimizer

Use the Experimentalist when the improvement belongs in the agent harness: the source that owns its workflow, tool use, state, and runtime behavior. It changes that source in candidate branches, evaluates each candidate against the agent's own harness, and can publish a validated winner for review.

Hand off to agents-optimize only when the request is explicitly limited to a deployed agent's routing, skills, prompts, or cost/latency, with no runtime implementation change. That workflow owns the deployment-tuning commands; do not substitute it for an Experimentalist source-change run.

Prepare the agent

The Experimentalist improves an existing, evaluable agent; it does not design or scaffold one. Before running it, make sure you have:

• agent source code and locked dependencies. For PR/MR publication, use a Git repository with a clean working tree and a pushed baseline revision;
• an ETHOS.md covering the agent's Goal, Scope, Tools, and Evaluation contract, including access configuration and constraints but never secret values;
• versioned, Harbor-compatible train and validation datasets. Keep a separate test split for a final manual baseline-versus-winner comparison. The Experimentalist profile and CLI accept only train and validation datasets, so run this third split manually with the agent's evaluator; never use it for candidate selection;
• a baseline smoke test that proves the agent runs, scores, and emits its expected artifacts and traces; and
• an optimizer.yaml profile, or the explicit inputs described below.

Record the agent revision, dataset versions, configuration, and result path with each run so another operator can reproduce it. If the agent or its Ethos does not exist yet, use nemo-explore to design it and nemo-ethos to create the Ethos before returning here.

Configure the environment

The repository-root .venv is the only supported Experimentalist environment. From the repository root, synchronize it and use its CLI:

uv sync
export NEMO="$PWD/.venv/bin/nemo"

Before running $NEMO or relying on a local platform, complete the repository-root SETUP.md. Then set the Platform URL and verify readiness:

export NMP_BASE_URL=http://localhost:8080
curl -sf "$NMP_BASE_URL/health/ready"

Stop if setup is incomplete or the readiness check fails.

The Experimentalist needs a running NeMo Platform, an LLM endpoint for its optimizer agents, and a model for each tier. For the NVIDIA Inference Gateway, set the inference key and model tiers:

export INFERENCE_API_KEY=<gateway-api-key>
export NEMO_EXPERIMENTALIST_MODELS_SMART=<model-name>
export NEMO_EXPERIMENTALIST_MODELS_MID=<model-name>
export NEMO_EXPERIMENTALIST_MODELS_FAST=<model-name>

Choose a strong frontier model for SMART: it performs the highest-stakes reasoning and code changes. Use a capable, lower-cost model for MID, where good judgment still matters but the work is less demanding. Use a fast, low-latency model for FAST; it serves high-volume supporting work and does not need the same reasoning depth.

The CLI uses the NVIDIA Inference Gateway by default and reuses INFERENCE_API_KEY as the Experimentalist key. For another OpenAI-compatible endpoint, set these instead of INFERENCE_API_KEY:

export NEMO_EXPERIMENTALIST_API_BASE=https://llm.example.com/v1
export NEMO_EXPERIMENTALIST_API_KEY=<endpoint-api-key>

Keep these values in your shell environment or an ignored environment file, never in optimizer.yaml, the run configuration, or Git. When a profile is loaded, the CLI automatically loads <profile-dir>/.env; existing shell values take precedence. Configure any credentials required by the agent under test according to its own setup. To publish a winning Git candidate, authenticate the relevant CLI in the same execution environment: gh auth login for GitHub or glab auth login for GitLab.

Isolate the run

The optimizer asks an LLM to edit the agent source and execute commands; Harbor also runs task containers. Do not run an unreviewed optimization with broad host access or production credentials. Prefer a disposable Docker Sandbox in clone mode, use dedicated, revocable, spending-limited keys, and retain only the network access the agent, Harbor, and model endpoint need. Clone mode protects the host checkout from writes but does not make ignored files secret: they remain readable in the sandbox. Follow the plugin's recommended laptop isolation when running locally.

From the repository root, create a clone-mode sandbox and run the Experimentalist inside it. Append the run-mode options from this skill to the last line:

repo="$(git rev-parse --show-toplevel)"
sbx create --clone --name nemo-experimentalist shell "$repo"
sbx exec --workdir "$repo" \
  --env UV_PROJECT_ENVIRONMENT=/home/agent/.venvs/nemo-platform \
  --env INFERENCE_API_KEY \
  --env NEMO_EXPERIMENTALIST_API_BASE \
  --env NEMO_EXPERIMENTALIST_API_KEY \
  --env NEMO_EXPERIMENTALIST_MODELS_SMART \
  --env NEMO_EXPERIMENTALIST_MODELS_MID \
  --env NEMO_EXPERIMENTALIST_MODELS_FAST \
  nemo-experimentalist \
  uv run --frozen --python 3.13 --package nemo-experimentalist-plugin --with ./plugins/nemo-agents \
  nemo agents experimentalist run

The sandbox has its own writable clone and virtual environment; do not point it at the host .venv. It is not a secret boundary: ignored files remain readable inside the sandbox. Forward only dedicated, revocable credentials and ensure the sandbox can reach the Platform, model endpoint, registry, and Harbor datasets it needs.

Pre-flight

Complete the SETUP.md and readiness check above before continuing. Run the built-in checks from the agent directory that contains optimizer.yaml, or pass the profile explicitly. Without a loaded profile, doctor cannot check the dataset or task-template artifacts:

$NEMO agents experimentalist doctor --profile path/to/optimizer.yaml

Stop and resolve any missing platform, credential, Docker, or dataset check before starting a run.

Choose a run mode

• Insight-driven: use an Insight ID or the default Insight written by nemo agents analyst run in the agent profile. An Insight is a systematic symptom inferred from a set of traces. Use this mode when the agent has no reliable, verifiable reward for the behavior you need to improve.
• Explicit evaluation: pass --no-insight, an agent, and separate train and validation datasets. Use this mode when a benchmark or other evaluator provides a trustworthy, verifiable reward that is a good measure of agent quality.

Keep validation independent from training. Never change an evaluation dataset just to make a candidate win.

Inputs

Choose either a reusable optimizer.yaml profile or pass the required inputs directly as CLI flags. A profile is optional; when used, flags override its values. With no --profile, the CLI discovers optimizer.yaml by walking up from the current directory.

An optimizer.yaml profile must identify the agent and its source, one task template, and independent train and validation datasets. It can also set the workspace, Ethos, run configuration, and framework skills. The Analyst and Experimentalist share this profile, allowing the latter to use the default Insight created by the former.

Explore configuration

Start with the deliberately small configuration below. The top-level options control rounds and candidate counts; source controls checkout behavior; storage controls candidate branches and PR/MR publication; outcome_evaluator_config controls trial execution; and eval_author controls Insight-driven evaluation authoring. See the example-agent walkthrough for a complete worked optimizer configuration and run.

To inspect the complete run schema—including nested options and their defaults—run this in the environment where the Experimentalist plugin is installed:

python -c \
  'import json; from nemo_experimentalist_plugin.config import EvolutionaryOptimizerConfig; print(json.dumps(EvolutionaryOptimizerConfig.model_json_schema(), indent=2))'

outcome_evaluator_config deliberately renders as an open mapping in that schema. For its Harbor-specific settings, use the smoke configuration below and the plugin's Harbor evaluator documentation; for example, outcome_evaluator_config.n_attempts defaults to 1.

Keep model endpoint and model-tier settings out of this file; configure those as Experimentalist deployment settings instead.

Tune the Evolutionary Optimizer

Use a smoke configuration to prove that source checkout, Harbor, credentials, and artifact collection work together. It is intentionally too small to judge whether an agent is better. For a real run, give the optimizer enough candidates and train tasks to learn, then rely on the separate validation split to select the winner.

A small explicit smoke configuration looks like this:

max_rounds: 1
min_rounds_before_stopping: 1
max_survivors: 1
max_candidates: 1
max_trajectory_tasks: 2
max_train_batch_tasks: 4
trajectory_scorer: null
terminator: null
storage:
  # The real default is true. Keep this false until you intend to create a PR/MR.
  publish_winner: false
builder_config:
  max_fix_attempts: 1
outcome_evaluator_config:
  n_attempts: 1
eval_author:
  max_traces: 3

builder_config.max_architecture_doc_iterations is how many iterations the builder can use to write architecture.md, and it defaults to 100. It is separate from the evaluation budget above, but each additional iteration adds model usage and run time to that step. Increase it when a run stops with Generation failed after 100 iterations (max_iterations=100). An agent that has many source files needs more iterations.

Create a low-cost smoke dataset

When the full dataset is expensive, create small copied train and validation subsets for the smoke run. Select a few representative task directories from each original split—for example, different task types or known failure modes—and copy each complete Harbor task directory, including its task.toml and verifier files:

mkdir -p smoke/train smoke/validation
for task in task-a task-b task-c; do
  cp -R "full/train/$task" smoke/train/
done
for task in task-d task-e; do
  cp -R "full/validation/$task" smoke/validation/
done

Point the smoke command at smoke/train and smoke/validation. Record the selected task IDs with the smoke configuration. Do not edit or replace the canonical splits, and do not use a smoke subset to decide whether a candidate is an improvement; run that decision against the full validation split.

For a real run, begin with the default values, raise only the settings that match the available evaluation budget, and preserve the same validation split throughout. Use storage.archive_candidates: true only when you need every candidate branch for review; it increases remote repository noise.

1. Run a smoke check

Use copied, low-cost dataset subsets and the smoke configuration above. Keep storage.publish_winner: false: its default is true, so a Git source can otherwise create a draft PR/MR when a candidate wins.

$NEMO agents experimentalist run \
  --no-insight \
  --agent path/to/agent \
  --train-dataset smoke/train \
  --validation-dataset smoke/validation \
  --config path/to/experimentalist-smoke.yaml

Run this command in a persistent terminal session such as tmux, because even the shortened walkthrough can take up to an hour. Redirect output to a log if you need to disconnect, then follow it with tail -f.

2. Run from an Insight

Before this mode, install the Experimentalist plugin, record agent traces in Intake, and run nemo agents analyst run to create the Insight. Use explicit evaluation instead if you already have a reliable reward and do not need an Insight.

Pass the inputs explicitly when running a one-off optimization:

$NEMO agents experimentalist run \
  --insight <platform-id-or-local-file> \
  --agent path-or-git-url \
  --task-template path/to/harbor-task-template \
  --train-dataset path-or-harbor-ref \
  --validation-dataset path-or-harbor-ref \
  --workspace <workspace> \
  --config path/to/experimentalist.yaml

--agent is optional when the Insight already identifies the agent source.

For a reusable setup, the equivalent information can live in an optimizer.yaml profile. When the profile is complete and the Analyst has created a default Insight for the agent, this shorter command is enough:

$NEMO agents experimentalist run --profile path/to/optimizer.yaml

Pass --insight <platform-id-or-local-file> to override the profile's default Insight.

3. Run from explicit datasets

$NEMO agents experimentalist run \
  --no-insight \
  --agent path-or-git-url \
  --ethos path/to/ETHOS.md \
  --train-dataset path-or-harbor-ref \
  --validation-dataset path-or-harbor-ref \
  --workspace <workspace> \
  --config path/to/experimentalist.yaml

For a Git source, append a ref such as git@github.com:owner/repository.git@main. The .git@ marker is required to select a ref; a URL such as https://github.com/owner/repository@main is treated as a repository URL with no ref and uses its default branch. For a local source, candidates remain in the experiment artifacts and no PR or MR can be opened.

4. Verify and review

Inspect <experiment-dir>/eval-and-optimize/run.json for the selected winner and compare candidate directories under <experiment-dir>/eval-and-optimize/agents/.

Review the validation result and generated PR or MR before merging. Do not overwrite the baseline agent directly.

If verification fails

• Fix every required doctor failure before retrying; run it with the same --profile and credentials as the optimization.
• If a run is interrupted, rerun with the same --experiment-dir. The optimizer detects completed rounds and resumes from its saved eval-and-optimize/run.json and artifacts.
• Do not reuse a partial experiment directory for a materially different source revision, dataset, or configuration; start a new directory instead.

Gotchas

• A smoke result proves the workflow, not an improvement. Select candidates on the full validation split and perform any third test-split comparison manually.
• Real runs are expensive and can run for hours. Bound rounds and task counts before starting, and inspect the generated winner before merging its PR/MR.