Nemo Guardrails

NeMo Guardrails CLI Reference

nemo guardrail (config CRUD + the /checks endpoint) and nemo inference virtual-models with a nemo-guardrails MiddlewareCall both back the same guardrail_config entity. The nemo guardrail chat completions create and nemo guardrail completions create commands are deprecated; do not use them.

Environment

• CLI paths: /app/.venv/bin/nemo and /app/.venv/bin/nmp (same binary, two names)
• API server: http://localhost:8080 (default, no auth needed for local services)
• Default workspace: default
• Workspace override: --workspace <name> flag, or NMP_WORKSPACE env var
• Entity ID: <workspace>/<model-entity-name>, e.g. default/meta-llama-3-70b-instruct. For backend models behind a provider, use the auto-discovered ID from served_models[].model_entity_id (see inference skill).
• Canned refusal: when a rail blocks, the response content is exactly "I'm sorry, I can't respond to that."

Prerequisites

• For nemo guardrail check alone: just a config and model entities reachable through IGW (so the main LLM and any task LLMs can resolve).
• For chat-time guardrailing on a VirtualModel: a real backend model entity the VirtualModel can proxy to. Use nemo-secrets and inference skills to register a secret + provider; the inference skill is the end-to-end reference for the provider/served-models/auto-discovery flow.

CLI gotchas

• Use nemo inference virtual-models (with hyphen), not nemo inference virtualmodels.
• Names are positional for nemo guardrail configs create <name> and nemo inference virtual-models create <name>.
• No nemo inference chat completions create command exists. Use nemo inference gateway model post v1/chat/completions <vm-name> (matches the inference skill).
• --data (configs) and --body / --models / --*-middleware (VirtualModels) take JSON strings. Watch shell quoting for embedded backticks/quotes — prefer --input-file for large payloads.
• config_type is always "guardrail_config" in a MiddlewareCall for nemo-guardrails.
• Each middleware list dispatches its own hook independently. If you want input rails AND output rails, the call must appear in both --request-middleware AND --response-middleware. See Failure cases.
• The guardrails field on a chat-completions request body is options-only on the IGW path (e.g. options.log, return_choice). It does not set the rails config — that comes from the VirtualModel's MiddlewareCall. This is the key behavior difference from the deprecated nemo guardrail chat completions create --guardrails '{"config_id":...}'.

Config CRUD Commands

# List configs in current workspace
nemo guardrail configs list

# Create a config (positional name + --data JSON)
nemo guardrail configs create <name> \
  --description "<description>" \
  --data '<json>'

# Retrieve a config by name
nemo guardrail configs get <name>

# Update a config (pass only fields to change)
nemo guardrail configs update <name> --description "<new description>"

# Delete a config
nemo guardrail configs delete <name>

For large config payloads, use --input-file config.json instead of inline --data.

Path A — nemo guardrail check

Evaluates a single message against a guardrail config server-side. The --guardrails '{"config_id": "..."}' argument is the resolver — the Guardrails service uses it to look up the config. --model is the main LLM. Self-check rails call this model; content-safety rails call any task LLMs the config references (for example, models[type=content_safety]), which must also be reachable through IGW.

nemo guardrail check \
  --model default/<main-model> \
  --messages '[{"role": "user", "content": "Your message here"}]' \
  --guardrails '{"config_id": "default/<config-name>"}' \
  --max-tokens 256

Path B — Guardrailed chat via IGW + VirtualModel

Configs are attached to a VirtualModel via a MiddlewareCall; chat completions then route through IGW which executes the plugin's input/output rails.

Step 1 — Provider + served models

Use the inference skill to set up a backend secret + provider and discover served entity IDs:

nemo inference providers get <provider-name> --workspace default \
  --output-format json | jq -r '.served_models[].model_entity_id'

Reference the auto-discovered entity ID in the VirtualModel --models array, and — if the rails config declares a task LLM via models[] — in models[].model too. The reconciler overwrites manually-registered entries.

Step 2 — Create the guardrail config

This example uses content-safety input and output rails. A content-safety rail asks a dedicated safety classifier to assess user input or model output against a safety taxonomy, then uses the parser's verdict to allow or block the turn. The flows call the model declared in models[] via $model=content_safety. The prompt includes the full unsafe-content category taxonomy used by the content-safety tutorial.

nemo guardrail configs create content-safety \
  --workspace default \
  --description "NemoGuard content-safety input and output rails" \
  --data '{
    "models": [
      {
        "type": "content_safety",
        "engine": "nim",
        "model": "default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3"
      }
    ],
    "rails": {
      "input": {"flows": ["content safety check input $model=content_safety"]},
      "output": {"flows": ["content safety check output $model=content_safety"]}
    },
    "prompts": [
      {
        "task": "content_safety_check_input $model=content_safety",
        "content": "Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a \"safe\" or \"unsafe\" rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.\n\n<BEGIN UNSAFE CONTENT CATEGORIES>\nS1: Violence.\nS2: Sexual.\nS3: Criminal Planning/Confessions.\nS4: Guns and Illegal Weapons.\nS5: Controlled/Regulated Substances.\nS6: Suicide and Self Harm.\nS7: Sexual (minor).\nS8: Hate/Identity Hate.\nS9: PII/Privacy.\nS10: Harassment.\nS11: Threat.\nS12: Profanity.\nS13: Needs Caution.\nS14: Other.\nS15: Manipulation.\nS16: Fraud/Deception.\nS17: Malware.\nS18: High Risk Gov Decision Making.\nS19: Political/Misinformation/Conspiracy.\nS20: Copyright/Trademark/Plagiarism.\nS21: Unauthorized Advice.\nS22: Illegal Activity.\nS23: Immoral/Unethical.\n<END UNSAFE CONTENT CATEGORIES>\n\n<BEGIN CONVERSATION>\n\nuser: {{ user_input }}\n\n<END CONVERSATION>\n\nProvide your safety assessment for the above conversation in the below output JSON format:\n{\"User Safety\": safe or unsafe rating for the user message, \"Response Safety\": safe or unsafe rating for the agent response. Omit if no agent response present. \"Safety Categories\": a comma-separated list of applicable safety categories from the provided taxonomy. Omit if all safe.}\n\nDo not include anything other than the output JSON in your response.\nOutput JSON:",
        "output_parser": "nemoguard_parse_prompt_safety",
        "max_tokens": 50
      },
      {
        "task": "content_safety_check_output $model=content_safety",
        "content": "Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a \"safe\" or \"unsafe\" rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.\n\n<BEGIN UNSAFE CONTENT CATEGORIES>\nS1: Violence.\nS2: Sexual.\nS3: Criminal Planning/Confessions.\nS4: Guns and Illegal Weapons.\nS5: Controlled/Regulated Substances.\nS6: Suicide and Self Harm.\nS7: Sexual (minor).\nS8: Hate/Identity Hate.\nS9: PII/Privacy.\nS10: Harassment.\nS11: Threat.\nS12: Profanity.\nS13: Needs Caution.\nS14: Other.\nS15: Manipulation.\nS16: Fraud/Deception.\nS17: Malware.\nS18: High Risk Gov Decision Making.\nS19: Political/Misinformation/Conspiracy.\nS20: Copyright/Trademark/Plagiarism.\nS21: Unauthorized Advice.\nS22: Illegal Activity.\nS23: Immoral/Unethical.\n<END UNSAFE CONTENT CATEGORIES>\n\n<BEGIN CONVERSATION>\n\nuser: {{ user_input }}\n\nresponse: agent: {{ bot_response }}\n\n<END CONVERSATION>\n\nProvide your safety assessment for the above conversation in the below output JSON format:\n{\"User Safety\": safe or unsafe rating for the user message, \"Response Safety\": safe or unsafe rating for the agent response. Omit if no agent response present. \"Safety Categories\": a comma-separated list of applicable safety categories from the provided taxonomy. Omit if all safe.}\n\nDo not include anything other than the output JSON in your response.\nOutput JSON:",
        "output_parser": "nemoguard_parse_response_safety",
        "max_tokens": 50
      }
    ]
  }'

The content_safety task model must be reachable through IGW. Use the auto-discovered entity ID from Step 1 and update models[].model if your workspace or provider uses a different entity name.

Step 3 — Create the VirtualModel with a guardrails MiddlewareCall

Three patterns, mirroring the input/output rail combinations.

Input + output rails — full content-safety coverage. Because the config created above declares both rails.input.flows and rails.output.flows, the call must appear in both lists; otherwise only the listed hook fires:

nemo inference virtual-models create vm-guarded-full \
  --workspace default \
  --models '[{"model":"default/<backend-model>","backend_format":"OPENAI_CHAT"}]' \
  --request-middleware '[{
    "name":"nemo-guardrails",
    "config_type":"guardrail_config",
    "config_id":"default/content-safety"
  }]' \
  --response-middleware '[{
    "name":"nemo-guardrails",
    "config_type":"guardrail_config",
    "config_id":"default/content-safety"
  }]'

Input rails only — block bad user prompts before they reach the backend:

nemo inference virtual-models create vm-guarded-input \
  --workspace default \
  --models '[{"model":"default/<backend-model>","backend_format":"OPENAI_CHAT"}]' \
  --request-middleware '[{
    "name":"nemo-guardrails",
    "config_type":"guardrail_config",
    "config_id":"default/content-safety"
  }]'

Output rails only — block bad bot responses:

nemo inference virtual-models create vm-guarded-output \
  --workspace default \
  --models '[{"model":"default/<backend-model>","backend_format":"OPENAI_CHAT"}]' \
  --response-middleware '[{
    "name":"nemo-guardrails",
    "config_type":"guardrail_config",
    "config_id":"default/content-safety"
  }]'

Inline config — no stored guardrail_config entity; the rails config goes directly inside the MiddlewareCall:

nemo inference virtual-models create vm-guarded-inline \
  --workspace default \
  --models '[{"model":"default/<backend-model>","backend_format":"OPENAI_CHAT"}]' \
  --response-middleware '[{
    "name":"nemo-guardrails",
    "config_type":"guardrail_config",
    "config":{"models":[...],"rails":{...},"prompts":[...]}
  }]'

The --models array is required and uses the backend model entity ID, not the VirtualModel name (the guardrails plugin doesn't route — the VirtualModel's default_model_entity / --models decide the upstream backend).

Step 4 — Make a guardrailed chat call

nemo inference gateway model post v1/chat/completions vm-guarded-full \
  --workspace default \
  --body '{
    "model":"default/<backend-model>",
    "messages":[{"role":"user","content":"Hello"}],
    "max_tokens":256
  }'

body["model"] is the backend entity ID and acts as the rails' main LLM. Content-safety flows call the task LLM from models[type=content_safety]; self-check flows, when used, call the main LLM. The VirtualModel name comes from the URL path.

For streaming, add "stream": true to the body — the plugin wraps the response iterator and runs output rails per chunk if rails.output.streaming.chunk_size is set in the config.

MiddlewareCall JSON Reference

MiddlewareCall is defined in packages/nemo_platform_plugin/src/nemo_platform_plugin/inference_middleware.py. For nemo-guardrails, the contract is:

{
  "name": "nemo-guardrails",
  "config_type": "guardrail_config",
  "config_id": "<workspace>/<config-name>"
}

Or, with an inline rails config:

{
  "name": "nemo-guardrails",
  "config_type": "guardrail_config",
  "config": {"models": [...], "rails": {...}, "prompts": [...]}
}

config_type is always the string "guardrail_config". Use config_id for stored entities (recommended — easier to update and reuse) and config for ephemeral inline configs.

Config JSON Structure

The main LLM is always the model from the chat request body (Path B) or the --model flag (Path A) — both Path A and Path B ignore any models[type=main] entry on the config and use the request's model instead. Self-check rails (self check input/self check output) call the main LLM, so they don't need a models[] entry at all. Add models[] only for task LLMs (content_safety, topic_control, jailbreak_detection, embeddings, vision_rails, etc.) that flows explicitly reference via $model=<type>.

Content safety with a task LLM

This pattern uses a dedicated content-safety classifier (for example, default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3) to assess both user input and model output against a safety taxonomy. The content_safety flows reference the task LLM via $model=content_safety, so the models[] entry is required here.

{
  "models": [
    {
      "type": "content_safety",
      "engine": "nim",
      "model": "default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3"
    }
  ],
  "rails": {
    "input":  {"flows": ["content safety check input $model=content_safety"]},
    "output": {"flows": ["content safety check output $model=content_safety"]}
  }
}

See Step 2 for the complete runnable config, including the full prompt taxonomy and output parsers.

Self-check rails

Self-check rails are most useful for tests and lightweight custom policies. For production safety, prefer content-safety rails with a dedicated classifier and taxonomy. Self-check runs against the main/general chat model and expects the model to answer Yes to block or No to allow. If the model returns anything else (for example, reasoning text, a refusal, or a truncated answer), the parser can default-deny and block the message. See Self-check example for a custom-policy example.

Streaming output rails

"output": {
  "flows": ["content safety check output $model=content_safety"],
  "streaming": {"enabled": true, "chunk_size": 200}
}

streaming is an OutputRailsStreamingConfig object, not a boolean. Other fields: enabled (default true), context_size (default 50), stream_first (default true). When unset, the plugin runs output rails on the assembled response only.

Field reference

models[] (optional — only for task LLMs)

• type — task to use the model for. Examples: "main", "content_safety", "topic_control", "jailbreak_detection", "embeddings", "vision_rails". Entries with type: "main" are ignored — omit them.
• engine — free-form string (commonly "nim"); the schema is permissive.
• model — <workspace>/<model-entity-name>. Must be reachable through IGW; use auto-discovered entity IDs from nemo inference providers get ... | jq '.served_models[].model_entity_id'.

Flows reference task LLMs by appending $model=<type> (e.g. content safety check input $model=content_safety). Self-check flows (self check input/self check output) run against the main LLM and need no models[] entry.

rails

• rails.input.flows: enables input rails (e.g. ["content safety check input $model=content_safety"])
• rails.output.flows: enables output rails (e.g. ["content safety check output $model=content_safety"])
• rails.output.streaming: optional {"chunk_size": N} for streaming output checks

prompts[]

• task: the task this prompt template is applied to. Common values: "self_check_input", "self_check_output", "content_safety_check_input $model=content_safety", "content_safety_check_output $model=content_safety". The $model=<type> suffix binds the prompt to a task LLM declared in models[].
• content: prompt template. Use {{ user_input }} to embed the user message and {{ bot_response }} to embed the model's response ({{ user_input }} is available in both input and output rail prompts; {{ bot_response }} is only meaningful in output rail prompts). For self-check tasks, end with a "Yes or No" question — the model answers Yes to block, No to allow. Content-safety tasks instead emit a JSON verdict consumed by output_parser.
• output_parser: optional parser name. Use this when the model returns a safety verdict that must be converted to an allow/block decision. is_content_safe is the generic parser for simple safe / unsafe or yes / no outputs. For content-safety checks using NemoGuard models, use "nemoguard_parse_prompt_safety" for the input rail and "nemoguard_parse_response_safety" for the output rail. For content-safety checks using Nemotron reasoning models, use "nemotron_reasoning_parse_prompt_safety" for the input rail and "nemotron_reasoning_parse_response_safety" for the output rail.

instructions[] (optional)

• type: "general"
• content: system-level context for the conversation.

Detecting Blocked Responses

When a rail triggers, the response content is exactly:

I'm sorry, I can't respond to that.

• Path A (nemo guardrail check): look for "status": "blocked" in the output JSON.
• Path B (chat completions through IGW): check response.choices[0].message.content == "I'm sorry, I can't respond to that.".

For streaming responses, the canned refusal arrives as a single non-streamed chunk when an output rail blocks mid-stream.

Self-check Example

Use self-check for tests or custom lightweight policies. For production safety, prefer the content-safety workflow above. These prompts run against the main chat model, so keep the prompt strict and prefer non-reasoning models for this pattern unless you also raise the prompt's max_tokens budget enough for the final Yes / No verdict.

nemo guardrail configs create custom-blocking \
  --description "Block fruit mentions in input, bread baking in output" \
  --data '{
    "rails":{
      "input":{"flows":["self check input"]},
      "output":{"flows":["self check output"]}
    },
    "prompts":[
      {"task":"self_check_input","content":"Check if the user message mentions any fruit (apple, banana, orange, grape, strawberry, mango, pear, peach, cherry, watermelon, lemon, lime, etc.).\n\nUser message: {{ user_input }}\n\nQuestion: Should the user message be blocked (Yes or No)?\nAnswer:"},
      {"task":"self_check_output","content":"Check if the bot response contains information about baking bread.\n\nBot response: {{ bot_response }}\n\nQuestion: Should the message be blocked (Yes or No)?\nAnswer:"}
    ],
    "instructions":[{"type":"general","content":"Below is a conversation between a user and a bot."}]
  }'

Then verify via nemo guardrail check, or attach to a VirtualModel as in Path B.

Failure cases

Plugin in only one of --request-middleware / --response-middleware

Each list dispatches its own hook independently (execute_request_middleware ↔ process_request, execute_response_middleware ↔ process_response — see services/core/inference-gateway/src/nmp/core/inference_gateway/api/middleware_registry.py).

If your config has both rails.input.flows and rails.output.flows but you attach the call to only one list, the unlisted side will silently no-op. Symptom: prompts pass through unblocked, or responses pass through unblocked, even though the config looks complete.

Fix: attach the same call to both --request-middleware and --response-middleware.

Putting guardrails field in the chat body expecting it to set the config

On the IGW VirtualModel path, the request body's guardrails dict is strictly an options envelope. Supported fields are options.log and return_choice. The rails config is resolved from the VirtualModel's MiddlewareCall, not from the request body.

Unsupported fields are rejected with a 422 validation error. This includes guardrails.config, guardrails.config_id, guardrails.options.rails, guardrails.options.llm_params, guardrails.options.llm_output, and guardrails.options.output_vars.

Symptom: you set --body '{"guardrails":{"config_id":"default/foo"},...}' expecting default/foo to be applied, but the request fails validation.

Fix: attach the config to the VirtualModel via --request-middleware / --response-middleware instead of trying to override per-request. Use guardrails.options.log only for diagnostic log fields and return_choice only for response formatting.

Task LLM (or main model) not addressable

If RailsConfig.models[].model references an entity that isn't in served_models, the rails will fail when they try to call that task LLM through IGW. Same applies to the main LLM (the request body's model on Path B, --model on Path A) — if it isn't reachable through IGW, self-check flows fail. Both usually show up as a 404 or timeout from the rails subsystem during the first request.

Fix: use auto-discovered entity IDs (from nemo inference providers get ... | jq '.served_models[].model_entity_id') both for the request/--model and for any task LLMs in models[]. See inference skill for the auto-discovery semantics.

VirtualModel 404 after working briefly

The provider reconciler re-syncs served_models from upstream auto-discovery every few seconds and drops manually-registered entries. See inference skill's "Reconciler-induced 404" failure case — the same fix applies (use auto-discovered entity IDs in --models).