NeMo Agent Hardener — war-game an agent
Agent Hardener attacks a sandboxed copy of an agent (garak), writes guardrails against what got through (defenders), and replays the attacks plus benign traffic to prove the fix (validators). Done means: a completed run with a scorecard — attacks blocked vs. benign false positives — and a hardened image/bundle you can adopt.
This skill is an interview: ask the user two questions (steps 2 and 3) and follow only the branch they pick. Do not dump every path at them.
Do not choose the target agent for the user. Even when the request names a concrete goal ("harden an example agent"), you must still ask which one in step 2 before any state-changing command (
agents register,agents package,agent-hardener init). Picking one yourself — because it is the default, the first listed, or the one you just built — is the single most common way this skill goes wrong. Ask first, every time.
1. Pre-flight
# `nemo` resolves its target from the active config context (~/.config/nhx/config.yaml)
# or an explicit NHX_BASE_URL. Do NOT export a localhost fallback here — it would
# override a configured remote. Leave NHX_BASE_URL unset to use your saved context;
# set it explicitly (export NHX_BASE_URL=http://localhost:8080) only to force a local
# platform for this shell when you have no local config context.
nemo agent-hardener doctor
doctorfailing on the venv → runnemo agent-hardener setuponce, thendoctoragain.- The platform must be running with the jobs controller (
nemo services run … --controllers jobs). Without it, Studio and job-driven runs sit increatedforever — silently. - Verify:
doctorreports every check green before moving on. If it cannot be made green, stop and tell the user what is missing; do not improvise around the sandbox.
2. Interview — which agent, and what kind?
Ask the user which agent to war-game — do not pick for them. Use an interactive question and offer, at minimum:
- each bundled example victim as a concrete choice — the plugin ships one per guardable harness
under
plugins/nemo-agent-hardener/examples/(relay-victimdeepagents,hermes-victimhermes,langchain-victim,langgraph-victim,other-victim); see that directory's README for the matrix; - an option to point you at their own directory (a Fabric
agent.yamlor a BYO Dockerfile project).
Once they choose, the agent's kind decides the route below — follow exactly one:
Route A — a Fabric agent (has an agent.yaml, harness deepagents or hermes)
Register it on the platform first, then point Agent Hardener at the registered agent. Packaging details (agent.yaml shape, Dockerfile contract): references/agent-packaging.md.
nemo agents package --agent agent.yaml --dockerfile ./Dockerfile --tag my-agent:v1
nemo agents register --name my-agent --agent-config agent.yaml
nemo agents deploy --agent my-agent --image my-agent:v1 # optional, but init reads the port from it
nemo agent-hardener init --agent my-agent --name my-agent
- Stage a clean directory before
create: it uploads everything next toagent.yaml, capped at ~900 KB / 500 files. Run state (.agent-hardener/, venvs) in that directory fails registration. - Verify:
initprints the derived egress.(none)for an agent with network MCP servers or a remote model is wrong — stop and investigate before running; the sandbox is default-deny and the agent's tools will silently fail mid-run.
Route B — your own image / framework (LangGraph, LangChain, DeepAgents library, …)
Bring-your-own mode: Agent Hardener war-games the image the user's own Dockerfile builds.
First, confirm NeMo Relay is attached — --relay-confirmed is a promise, not proof, and a
victim that looks instrumented but runs unguarded corrupts the whole run. Walk
references/relay-attachment.md; if the agent is not instrumented
yet, help the user make the edits it describes (and defer to NeMo Relay's own documentation and
published skills for frameworks beyond those).
nemo agent-hardener init --project-dir ./my-agent --name my-agent --harness langgraph --relay-confirmed
- Everything the Dockerfile states (port, start command, env, secret names) is derived. The
command fails naming exactly what it could not derive — supply those flags (
--start-command,--dockerfile,--secrets,--egress, …) and re-run. - Secrets and egress usually cannot be derived: pass
--secrets NAMEfor each credential and--egress host[:port]for each host the agent legitimately reaches (a bare host opens 443 only). - Verify:
initwrites the manifest and echoes the derived facts; read them back to the user and confirm they match the agent before continuing.
Route C — a claude / codex gateway agent
Stop honestly: Agent Hardener refuses these at init by design. They run behind a compiled gateway
with no in-process interception point, so a guardrail could never actually block a tool call.
Suggest the garak skill for scan-only coverage.
3. Interview — Studio or CLI?
Studio
- Collect credentials first: store each secret the agent needs with the platform
nemo-secretsskill (ornemo secretsCLI) so the run can reference them. - Open
$NHX_BASE_URL/studio/, pick the workspace, choose Agent Hardener in the nav. - Manifests → New manifest — the source toggle offers Registered agent and Bring your own (BYO takes a project archive upload).
- Open the manifest page → Run war-game → fill env vars → submit.
- Verify: the run appears on the runs list and its swarm lanes (attacker / sandbox / defender /
validator) start moving. A run stuck in
createdmeans the jobs controller is missing (Pre-flight).
CLI
Continue with steps 4–7 below.
4. Synthesize the benign suite (CLI)
The benign suite is what proves the defenders did not break the agent for real users. It is required before a first run, and this is also the first victim bring-up — Relay problems surface here, early and cheap.
Do not run this command yourself — hand it to the user. It is interactive by default: it
interviews them about realistic traffic, and only they can answer for their users. Ask them to run
it in this session with the ! prefix, and wait for it to finish before continuing:
!nemo agent-hardener synth-benign --manifest-id my-agent --env-file ./secrets.env
Only run it yourself if the user explicitly asks you to, or if they want the interview skipped:
--yes keeps the interview but auto-accepts each recommended default; --no-interactive skips it
entirely (leaner rules-only suite, for CI).
- Verify: it ends with
Cached N benign requests. AVictimNotInstrumentedErrorhere means Relay is not attached → references/relay-attachment.md.
5. Run the war-game (CLI)
nemo agent-hardener run --manifest-id my-agent --env-file ./secrets.env
- Fast path for iterating on defenses — replay a previous run's hits instead of re-attacking:
--replay-hitlog <workspace>/<fileset>. That is a fileset reference, not a file path; create the fileset first (nemo files filesets create …) and upload into it (nemo files upload …does not auto-create a named fileset). - One war-game per host: every victim binds port 8000. If a second run is needed, wait or use
--port. - Verify:
nemo agent-hardener status --limit 5shows the run progressing through rounds. A run that endsvalidation_failedis a result — some attacks still got through — not a crash; read the scorecard.
6. Read the results
The run directory (printed at the end, under the manifest's .agent-hardener/ state) holds:
- the scorecard — attacks blocked, benign false positives, per-round detail;
plugins.toml— the guardrails the defenders wrote;mitigations.json— machine-readable mitigation records;- the hardened image/bundle — the user's image with the guardrails delivered into it.
Report the scorecard numbers to the user plainly, including what was not blocked.
7. Adopt the hardened agent
For a registered agent, deploy the hardened image the run produced:
nemo agents deploy --agent my-agent --mode docker --image my-agent-hardened:<run_id>
--image is rejected in the default subprocess mode — --mode docker is required. For a BYO
agent, hand the user the hardened bundle location; shipping it is theirs.
- Verify:
nemo agents deployments listshows the new deployment, and a smoke request answers.
If verification fails
Match the symptom in references/troubleshooting.md — it maps each failure we have actually hit (stuck jobs, OOM-killed victims, timeouts, port conflicts, fileset refs) to its cause and fix. Do not report a step done that you could not verify; surface what you saw and ask the user.
Gotchas
- Manifests are frozen targets: after the agent changes,
nemo agent-hardener refresh --manifest-id my-agentre-resolves it (Studio: Refresh Target). Runs log a warning when the agent is newer than the manifest. - A host service the victim calls (e.g. a remote MCP server) is declared at init with
--backend NAME:PORT[,PORT](repeatable) and reached viahost.docker.internal. The gateway backend is added automatically (its name is reserved); declared backends surviverefresh. --env-filevalues are merged into the victim's.envand win over the operator env.- A failed run keeps its hitlog, so it can be replayed with
--replay-hitlog. nemo agent-hardener manifest show my-agent/manifest setread and tweak the stored config; model-group overrides (--attack-model,--attack-key-secret,--analysis-model, …) live onrun.- Env vars vs secrets:
--env KEY=VALUEis for plain config; anything credential-shaped goes through--secrets NAME=workspace/secret(read from the Secrets store at run time) or--env-file, so it never lands in the manifest. A deployed agent's own secret references are recorded for you. - A key exported in the shell that started the platform overrides
~/.agent-hardener/.env; the run logsthe platform environment overrides <NAME>when that happens. - Running the platform from a source checkout needs
NEMO_AGENTS_ALLOW_UNPUBLISHED_CONTRACT_VERSION=1in the platform's environment.