# Nemo Guardrails > NeMo guardrails CLI reference for creating guardrail configs and applying them to chat completions through an IGW VirtualModel. Covers config CRUD, the standalone `/checks` endpoint for verifying a config, and the `nemo-guardrails` inference middleware plugin attached via VirtualModel MiddlewareCalls. Use when the task involves guardrail configurations, content safety, input/output rails, or `nemo guardrail` / `nemo inference virtual-models` CLI commands for guardrailing inference. - Skill: `nvidia-nemo/nemo-guardrails` (Agent Skill) - Install (CLI): `npx skillmds@latest add nvidia-nemo/nemo-guardrails` - Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-nemo/nemo-guardrails/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: NVIDIA NeMo (https://skillmd.com/u/nvidia-nemo) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/nvidia-nemo/nemo-guardrails --- # NeMo Guardrails CLI Reference `nemo guardrail` (config CRUD + the `/checks` endpoint) and `nemo inference virtual-models` with a `nemo-guardrails` MiddlewareCall both back the same `guardrail_config` entity. The `nemo guardrail chat completions create` and `nemo guardrail completions create` commands are deprecated; do not use them. ## Environment - **CLI paths**: `/app/.venv/bin/nemo` and `/app/.venv/bin/nmp` (same binary, two names) - **API server**: `http://localhost:8080` (default, no auth needed for local services) - **Default workspace**: `default` - **Workspace override**: `--workspace ` flag, or `NMP_WORKSPACE` env var - **Entity ID**: `/`, e.g. `default/meta-llama-3-70b-instruct`. For backend models behind a provider, use the auto-discovered ID from `served_models[].model_entity_id` (see `inference` skill). - **Canned refusal**: when a rail blocks, the response content is exactly `"I'm sorry, I can't respond to that."` ## Prerequisites - For **`nemo guardrail check`** alone: just a config and model entities reachable through IGW (so the main LLM and any task LLMs can resolve). - For **chat-time guardrailing on a VirtualModel**: a real backend model entity the VirtualModel can proxy to. Use `nemo-secrets` and `inference` skills to register a secret + provider; the `inference` skill is the end-to-end reference for the provider/served-models/auto-discovery flow. --- ## CLI gotchas - **Use `nemo inference virtual-models` (with hyphen)**, not `nemo inference virtualmodels`. - **Names are positional** for `nemo guardrail configs create ` and `nemo inference virtual-models create `. - **No `nemo inference chat completions create` command exists.** Use `nemo inference gateway model post v1/chat/completions ` (matches the `inference` skill). - **`--data` (configs) and `--body` / `--models` / `--*-middleware` (VirtualModels) take JSON strings.** Watch shell quoting for embedded backticks/quotes — prefer `--input-file` for large payloads. - **`config_type` is always `"guardrail_config"`** in a `MiddlewareCall` for `nemo-guardrails`. - **Each middleware list dispatches its own hook independently.** If you want input rails AND output rails, the call must appear in **both** `--request-middleware` AND `--response-middleware`. See `Failure cases`. - **The `guardrails` field on a chat-completions request body is options-only on the IGW path** (e.g. `options.log`, `return_choice`). It does **not** set the rails config — that comes from the VirtualModel's `MiddlewareCall`. This is the key behavior difference from the deprecated `nemo guardrail chat completions create --guardrails '{"config_id":...}'`. --- ## Config CRUD Commands ```bash # List configs in current workspace nemo guardrail configs list # Create a config (positional name + --data JSON) nemo guardrail configs create \ --description "" \ --data '' # Retrieve a config by name nemo guardrail configs get # Update a config (pass only fields to change) nemo guardrail configs update --description "" # Delete a config nemo guardrail configs delete ``` For large config payloads, use `--input-file config.json` instead of inline `--data`. --- ## Path A — `nemo guardrail check` Evaluates a single message against a guardrail config server-side. The `--guardrails '{"config_id": "..."}'` argument is the resolver — the Guardrails service uses it to look up the config. `--model` is the **main** LLM. Self-check rails call this model; content-safety rails call any task LLMs the config references (for example, `models[type=content_safety]`), which must also be reachable through IGW. ```bash nemo guardrail check \ --model default/ \ --messages '[{"role": "user", "content": "Your message here"}]' \ --guardrails '{"config_id": "default/"}' \ --max-tokens 256 ``` --- ## Path B — Guardrailed chat via IGW + VirtualModel Configs are attached to a VirtualModel via a `MiddlewareCall`; chat completions then route through IGW which executes the plugin's input/output rails. ### Step 1 — Provider + served models Use the `inference` skill to set up a backend secret + provider and discover served entity IDs: ```bash nemo inference providers get --workspace default \ --output-format json | jq -r '.served_models[].model_entity_id' ``` Reference the auto-discovered entity ID in the VirtualModel `--models` array, and — if the rails config declares a task LLM via `models[]` — in `models[].model` too. The reconciler overwrites manually-registered entries. ### Step 2 — Create the guardrail config This example uses content-safety input and output rails. A content-safety rail asks a dedicated safety classifier to assess user input or model output against a safety taxonomy, then uses the parser's verdict to allow or block the turn. The flows call the model declared in `models[]` via `$model=content_safety`. The prompt includes the full unsafe-content category taxonomy used by the content-safety tutorial. ```bash nemo guardrail configs create content-safety \ --workspace default \ --description "NemoGuard content-safety input and output rails" \ --data '{ "models": [ { "type": "content_safety", "engine": "nim", "model": "default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3" } ], "rails": { "input": {"flows": ["content safety check input $model=content_safety"]}, "output": {"flows": ["content safety check output $model=content_safety"]} }, "prompts": [ { "task": "content_safety_check_input $model=content_safety", "content": "Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a \"safe\" or \"unsafe\" rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.\n\n\nS1: Violence.\nS2: Sexual.\nS3: Criminal Planning/Confessions.\nS4: Guns and Illegal Weapons.\nS5: Controlled/Regulated Substances.\nS6: Suicide and Self Harm.\nS7: Sexual (minor).\nS8: Hate/Identity Hate.\nS9: PII/Privacy.\nS10: Harassment.\nS11: Threat.\nS12: Profanity.\nS13: Needs Caution.\nS14: Other.\nS15: Manipulation.\nS16: Fraud/Deception.\nS17: Malware.\nS18: High Risk Gov Decision Making.\nS19: Political/Misinformation/Conspiracy.\nS20: Copyright/Trademark/Plagiarism.\nS21: Unauthorized Advice.\nS22: Illegal Activity.\nS23: Immoral/Unethical.\n\n\n\n\nuser: {{ user_input }}\n\n\n\nProvide your safety assessment for the above conversation in the below output JSON format:\n{\"User Safety\": safe or unsafe rating for the user message, \"Response Safety\": safe or unsafe rating for the agent response. Omit if no agent response present. \"Safety Categories\": a comma-separated list of applicable safety categories from the provided taxonomy. Omit if all safe.}\n\nDo not include anything other than the output JSON in your response.\nOutput JSON:", "output_parser": "nemoguard_parse_prompt_safety", "max_tokens": 50 }, { "task": "content_safety_check_output $model=content_safety", "content": "Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a \"safe\" or \"unsafe\" rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.\n\n\nS1: Violence.\nS2: Sexual.\nS3: Criminal Planning/Confessions.\nS4: Guns and Illegal Weapons.\nS5: Controlled/Regulated Substances.\nS6: Suicide and Self Harm.\nS7: Sexual (minor).\nS8: Hate/Identity Hate.\nS9: PII/Privacy.\nS10: Harassment.\nS11: Threat.\nS12: Profanity.\nS13: Needs Caution.\nS14: Other.\nS15: Manipulation.\nS16: Fraud/Deception.\nS17: Malware.\nS18: High Risk Gov Decision Making.\nS19: Political/Misinformation/Conspiracy.\nS20: Copyright/Trademark/Plagiarism.\nS21: Unauthorized Advice.\nS22: Illegal Activity.\nS23: Immoral/Unethical.\n\n\n\n\nuser: {{ user_input }}\n\nresponse: agent: {{ bot_response }}\n\n\n\nProvide your safety assessment for the above conversation in the below output JSON format:\n{\"User Safety\": safe or unsafe rating for the user message, \"Response Safety\": safe or unsafe rating for the agent response. Omit if no agent response present. \"Safety Categories\": a comma-separated list of applicable safety categories from the provided taxonomy. Omit if all safe.}\n\nDo not include anything other than the output JSON in your response.\nOutput JSON:", "output_parser": "nemoguard_parse_response_safety", "max_tokens": 50 } ] }' ``` The `content_safety` task model must be reachable through IGW. Use the auto-discovered entity ID from Step 1 and update `models[].model` if your workspace or provider uses a different entity name. ### Step 3 — Create the VirtualModel with a guardrails MiddlewareCall Three patterns, mirroring the input/output rail combinations. **Input + output rails** — full content-safety coverage. Because the config created above declares both `rails.input.flows` and `rails.output.flows`, the call must appear in **both** lists; otherwise only the listed hook fires: ```bash nemo inference virtual-models create vm-guarded-full \ --workspace default \ --models '[{"model":"default/","backend_format":"OPENAI_CHAT"}]' \ --request-middleware '[{ "name":"nemo-guardrails", "config_type":"guardrail_config", "config_id":"default/content-safety" }]' \ --response-middleware '[{ "name":"nemo-guardrails", "config_type":"guardrail_config", "config_id":"default/content-safety" }]' ``` **Input rails only** — block bad user prompts before they reach the backend: ```bash nemo inference virtual-models create vm-guarded-input \ --workspace default \ --models '[{"model":"default/","backend_format":"OPENAI_CHAT"}]' \ --request-middleware '[{ "name":"nemo-guardrails", "config_type":"guardrail_config", "config_id":"default/content-safety" }]' ``` **Output rails only** — block bad bot responses: ```bash nemo inference virtual-models create vm-guarded-output \ --workspace default \ --models '[{"model":"default/","backend_format":"OPENAI_CHAT"}]' \ --response-middleware '[{ "name":"nemo-guardrails", "config_type":"guardrail_config", "config_id":"default/content-safety" }]' ``` **Inline config** — no stored `guardrail_config` entity; the rails config goes directly inside the `MiddlewareCall`: ```bash nemo inference virtual-models create vm-guarded-inline \ --workspace default \ --models '[{"model":"default/","backend_format":"OPENAI_CHAT"}]' \ --response-middleware '[{ "name":"nemo-guardrails", "config_type":"guardrail_config", "config":{"models":[...],"rails":{...},"prompts":[...]} }]' ``` The `--models` array is required and uses the backend model entity ID, not the VirtualModel name (the guardrails plugin doesn't route — the VirtualModel's `default_model_entity` / `--models` decide the upstream backend). ### Step 4 — Make a guardrailed chat call ```bash nemo inference gateway model post v1/chat/completions vm-guarded-full \ --workspace default \ --body '{ "model":"default/", "messages":[{"role":"user","content":"Hello"}], "max_tokens":256 }' ``` `body["model"]` is the backend entity ID and acts as the rails' **main** LLM. Content-safety flows call the task LLM from `models[type=content_safety]`; self-check flows, when used, call the main LLM. The VirtualModel name comes from the URL path. For streaming, add `"stream": true` to the body — the plugin wraps the response iterator and runs output rails per chunk if `rails.output.streaming.chunk_size` is set in the config. --- ## MiddlewareCall JSON Reference `MiddlewareCall` is defined in [packages/nemo_platform_plugin/src/nemo_platform_plugin/inference_middleware.py](packages/nemo_platform_plugin/src/nemo_platform_plugin/inference_middleware.py). For `nemo-guardrails`, the contract is: ```json { "name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "/" } ``` Or, with an inline rails config: ```json { "name": "nemo-guardrails", "config_type": "guardrail_config", "config": {"models": [...], "rails": {...}, "prompts": [...]} } ``` `config_type` is always the string `"guardrail_config"`. Use `config_id` for stored entities (recommended — easier to update and reuse) and `config` for ephemeral inline configs. --- ## Config JSON Structure The **main** LLM is always the model from the chat request body (Path B) or the `--model` flag (Path A) — both Path A and Path B ignore any `models[type=main]` entry on the config and use the request's model instead. Self-check rails (`self check input`/`self check output`) call the main LLM, so they don't need a `models[]` entry at all. Add `models[]` only for **task LLMs** (`content_safety`, `topic_control`, `jailbreak_detection`, `embeddings`, `vision_rails`, etc.) that flows explicitly reference via `$model=`. ### Content safety with a task LLM This pattern uses a dedicated content-safety classifier (for example, `default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3`) to assess both user input and model output against a safety taxonomy. The `content_safety` flows reference the task LLM via `$model=content_safety`, so the `models[]` entry **is** required here. ```json { "models": [ { "type": "content_safety", "engine": "nim", "model": "default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3" } ], "rails": { "input": {"flows": ["content safety check input $model=content_safety"]}, "output": {"flows": ["content safety check output $model=content_safety"]} } } ``` See [Step 2](#step-2--create-the-guardrail-config) for the complete runnable config, including the full prompt taxonomy and output parsers. ### Self-check rails Self-check rails are most useful for tests and lightweight custom policies. For production safety, prefer content-safety rails with a dedicated classifier and taxonomy. Self-check runs against the main/general chat model and expects the model to answer `Yes` to block or `No` to allow. If the model returns anything else (for example, reasoning text, a refusal, or a truncated answer), the parser can default-deny and block the message. See [Self-check example](#self-check-example) for a custom-policy example. ### Streaming output rails ```json "output": { "flows": ["content safety check output $model=content_safety"], "streaming": {"enabled": true, "chunk_size": 200} } ``` `streaming` is an `OutputRailsStreamingConfig` object, not a boolean. Other fields: `enabled` (default `true`), `context_size` (default `50`), `stream_first` (default `true`). When unset, the plugin runs output rails on the assembled response only. ### Field reference **models[]** (optional — only for task LLMs) - `type` — task to use the model for. Examples: `"main"`, `"content_safety"`, `"topic_control"`, `"jailbreak_detection"`, `"embeddings"`, `"vision_rails"`. Entries with `type: "main"` are ignored — omit them. - `engine` — free-form string (commonly `"nim"`); the schema is permissive. - `model` — `/`. Must be reachable through IGW; use auto-discovered entity IDs from `nemo inference providers get ... | jq '.served_models[].model_entity_id'`. Flows reference task LLMs by appending `$model=` (e.g. `content safety check input $model=content_safety`). Self-check flows (`self check input`/`self check output`) run against the main LLM and need no `models[]` entry. **rails** - `rails.input.flows`: enables input rails (e.g. `["content safety check input $model=content_safety"]`) - `rails.output.flows`: enables output rails (e.g. `["content safety check output $model=content_safety"]`) - `rails.output.streaming`: optional `{"chunk_size": N}` for streaming output checks **prompts[]** - `task`: the task this prompt template is applied to. Common values: `"self_check_input"`, `"self_check_output"`, `"content_safety_check_input $model=content_safety"`, `"content_safety_check_output $model=content_safety"`. The `$model=` suffix binds the prompt to a task LLM declared in `models[]`. - `content`: prompt template. Use `{{ user_input }}` to embed the user message and `{{ bot_response }}` to embed the model's response (`{{ user_input }}` is available in both input and output rail prompts; `{{ bot_response }}` is only meaningful in output rail prompts). For self-check tasks, end with a "Yes or No" question — the model answers `Yes` to block, `No` to allow. Content-safety tasks instead emit a JSON verdict consumed by `output_parser`. - `output_parser`: optional parser name. Use this when the model returns a safety verdict that must be converted to an allow/block decision. `is_content_safe` is the generic parser for simple `safe` / `unsafe` or `yes` / `no` outputs. For content-safety checks using NemoGuard models, use `"nemoguard_parse_prompt_safety"` for the input rail and `"nemoguard_parse_response_safety"` for the output rail. For content-safety checks using Nemotron reasoning models, use `"nemotron_reasoning_parse_prompt_safety"` for the input rail and `"nemotron_reasoning_parse_response_safety"` for the output rail. **instructions[]** (optional) - `type`: `"general"` - `content`: system-level context for the conversation. --- ## Detecting Blocked Responses When a rail triggers, the response content is exactly: ``` I'm sorry, I can't respond to that. ``` - **Path A (`nemo guardrail check`)**: look for `"status": "blocked"` in the output JSON. - **Path B (chat completions through IGW)**: check `response.choices[0].message.content == "I'm sorry, I can't respond to that."`. For streaming responses, the canned refusal arrives as a single non-streamed chunk when an output rail blocks mid-stream. --- ## Self-check Example Use self-check for tests or custom lightweight policies. For production safety, prefer the content-safety workflow above. These prompts run against the main chat model, so keep the prompt strict and prefer non-reasoning models for this pattern unless you also raise the prompt's `max_tokens` budget enough for the final `Yes` / `No` verdict. ```bash nemo guardrail configs create custom-blocking \ --description "Block fruit mentions in input, bread baking in output" \ --data '{ "rails":{ "input":{"flows":["self check input"]}, "output":{"flows":["self check output"]} }, "prompts":[ {"task":"self_check_input","content":"Check if the user message mentions any fruit (apple, banana, orange, grape, strawberry, mango, pear, peach, cherry, watermelon, lemon, lime, etc.).\n\nUser message: {{ user_input }}\n\nQuestion: Should the user message be blocked (Yes or No)?\nAnswer:"}, {"task":"self_check_output","content":"Check if the bot response contains information about baking bread.\n\nBot response: {{ bot_response }}\n\nQuestion: Should the message be blocked (Yes or No)?\nAnswer:"} ], "instructions":[{"type":"general","content":"Below is a conversation between a user and a bot."}] }' ``` Then verify via `nemo guardrail check`, or attach to a VirtualModel as in [Path B](#path-b--guardrailed-chat-via-igw--virtualmodel). --- ## Failure cases ### Plugin in only one of `--request-middleware` / `--response-middleware` Each list dispatches its own hook independently (`execute_request_middleware` ↔ `process_request`, `execute_response_middleware` ↔ `process_response` — see [services/core/inference-gateway/src/nmp/core/inference_gateway/api/middleware_registry.py](services/core/inference-gateway/src/nmp/core/inference_gateway/api/middleware_registry.py)). If your config has both `rails.input.flows` and `rails.output.flows` but you attach the call to only one list, the unlisted side will silently no-op. Symptom: prompts pass through unblocked, or responses pass through unblocked, even though the config looks complete. Fix: attach the same call to both `--request-middleware` and `--response-middleware`. ### Putting `guardrails` field in the chat body expecting it to set the config On the **IGW VirtualModel path**, the request body's `guardrails` dict is strictly an options envelope. Supported fields are `options.log` and `return_choice`. The rails config is resolved from the VirtualModel's `MiddlewareCall`, not from the request body. Unsupported fields are rejected with a `422` validation error. This includes `guardrails.config`, `guardrails.config_id`, `guardrails.options.rails`, `guardrails.options.llm_params`, `guardrails.options.llm_output`, and `guardrails.options.output_vars`. Symptom: you set `--body '{"guardrails":{"config_id":"default/foo"},...}'` expecting `default/foo` to be applied, but the request fails validation. Fix: attach the config to the VirtualModel via `--request-middleware` / `--response-middleware` instead of trying to override per-request. Use `guardrails.options.log` only for diagnostic log fields and `return_choice` only for response formatting. ### Task LLM (or main model) not addressable If `RailsConfig.models[].model` references an entity that isn't in `served_models`, the rails will fail when they try to call that task LLM through IGW. Same applies to the main LLM (the request body's model on Path B, `--model` on Path A) — if it isn't reachable through IGW, self-check flows fail. Both usually show up as a 404 or timeout from the rails subsystem during the first request. Fix: use auto-discovered entity IDs (from `nemo inference providers get ... | jq '.served_models[].model_entity_id'`) both for the request/`--model` and for any task LLMs in `models[]`. See `inference` skill for the auto-discovery semantics. ### VirtualModel 404 after working briefly The provider reconciler re-syncs `served_models` from upstream auto-discovery every few seconds and drops manually-registered entries. See `inference` skill's "Reconciler-induced 404" failure case — the same fix applies (use auto-discovered entity IDs in `--models`).