NVIDIA Inference Builder
A code generator for Vision AI pipelines with high-performance video and
streaming capabilities on NVIDIA GPUs. It turns a YAML config (+ optional
OpenAPI spec and custom processors) into a deployable inference pipeline,
packaged as a Docker container or standalone application.
Setup
Inference Builder is part of the DeepStream mono-repo. Resolve it from the
repository root so the skill works from any subdirectory:
DEEPSTREAM_ROOT="$(git rev-parse --show-toplevel)"
INFERENCE_BUILDER_ROOT="$DEEPSTREAM_ROOT/tools/inference_builder"
source "$INFERENCE_BUILDER_ROOT/.venv/bin/activate"
python "$INFERENCE_BUILDER_ROOT/builder/main.py" --help
First-time venv setup
cd "$INFERENCE_BUILDER_ROOT"
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
Getting current CLI usage
Always run with --help for authoritative flags — do NOT rely on memorized
syntax:
python "$INFERENCE_BUILDER_ROOT/builder/main.py" --help
All paths below are relative to tools/inference_builder/ unless stated otherwise.
Key Directories
| Path |
What lives there |
builder/samples/ |
Working examples for every backend — read builder/samples/README.md for the full index, then start from the closest matching example |
schemas/config.schema.json |
Main config schema — all fields, data types, and validation rules |
schemas/README.md |
Comprehensive reference: backends, parameters, data types, preprocessors, responders, routing |
schemas/backends/ |
Per-backend schemas and parameter definitions |
templates/backend/ |
Jinja templates for each inference backend |
templates/responder/ |
Jinja templates for server endpoint handlers |
lib/ |
Shared runtime libraries included in generated pipelines |
builder/tests/ |
Integration test framework (see run_tests.sh --help) |
Instructions
- Browse samples & schemas — find the closest match in
builder/samples/;
consult schemas/config.schema.json and schemas/backends/ for field reference
- Write YAML config — validate against the schema; consult
schemas/README.md for data types, preprocessors, responders, and routing.
Prefer choices that keep the hot path on GPU and skip avoidable GPU↔CPU
round-trips; prefer GPU acceleration across the whole pipeline — decode/
encode, scale & color convert, batch, preprocess, infer, track, and overlay
- Generate pipeline —
python builder/main.py <config.yaml> [options]
- Prepare model repository — download models from NGC/HuggingFace into
the model_repo directory
- Generate nvinfer config (DeepStream only) — check if the downloaded
model already includes runtime config files (nvdsinfer_config.yaml,
nvdspreprocess_config.yaml); if not, generate them and place alongside
the model. Skip this step entirely for non-DeepStream backends
- Select platform & build Docker image — read
doc/platform-guide.md to
choose the correct base image and Dockerfile for the target hardware
(x86_64, Jetson/Tegra, DGX Spark); then build the image
- Run & test — run the container with the model repository mounted;
for serverless pipelines the container runs to completion; for HTTP
servers (fastapi/triton/nim) send test requests to verify. If GPU
resources are available, perform an end-to-end smoke test against the
running container to validate the full pipeline on real hardware
Non-Obvious Knowledge
- If the user's model request is ambiguous (e.g., unclear model variant,
version, precision, or source), always ask for clarification before
proceeding — do not guess.
- Before using any model, gather as much information as possible about it:
version, architecture, input/output shapes, precision, and expected
preprocessing. Cross-check this information against the pipeline YAML
config and any runtime config (e.g., nvdsinfer_config.yaml) to ensure
consistency — mismatches in input dims, scale factors, or label files
are common sources of silent failures.
- Config YAML supports runtime env-var substitution:
$ENV_VAR|default_value
(auto-typed based on the default).
- Server type
serverless does NOT need an OpenAPI spec or server definition.
- Custom processor class
name attribute must exactly match the processor
name in the YAML config.
- Default to bridge networking plus explicit
-p port mappings. Recommend
Docker host-network or host-IPC modes only when the deployment explicitly
needs host namespace sharing, and call out the security tradeoff before
opting in.
- Before running containers, check
nvidia-smi for free GPU memory. If GPUs
are occupied or low on memory, select a free GPU with --gpus device=N or
ask the user to free resources — OOM failures can be silent or misleading.
- Key runtime env vars:
HTTP_PORT (default 8000), LOG_LEVEL (default INFO),
DEBUG (0/1), N_CODEC_INSTANCES (parallel video decoders),
MAX_BATCH_SIZE (live stream frame batching).
- DeepStream backends require a separate
nvdsinfer_config.yaml in the
model directory.
- When writing new configs, the required top-level fields are only
name,
model_repo, and models — everything else is optional.
Examples
- "Generate an inference-builder YAML config for a DeepStream detection service
that reads RTSP, uses Triton, and exposes a FastAPI endpoint."
- "Adapt the dummy sample into a serverless image-inference pipeline and show
the Docker build and run commands."
1---2name: inference-builder3description: |4 Generate deployable Vision AI pipelines with high-performance video and5 streaming capabilities on NVIDIA GPUs. Activate when users want to: create6 GPU-accelerated inference microservices or standalone apps for vision,7 video, or streaming workloads; write or edit pipeline YAML configs; build8 Docker images for GPU inference; work with models from NGC or HuggingFace;9 or deploy with DeepStream, Triton, vLLM, TensorRT-LLM, or PyTorch backends.10owner: "chunlinl@nvidia.com"11metadata:12 source: https://skillmd.com/skills/nvidia/inference-builder13 author: "Chunlin Li <chunlinl@nvidia.com>"14 version: "1.0.0"15 tags:16 - inference17 - nvidia18 - deepstream19 - vllm20 - tensorrt-llm21 - triton22 - docker23 - gpu24service: "inference"25reviewed: "2026-04-09"26---2728# NVIDIA Inference Builder2930A code generator for Vision AI pipelines with high-performance video and31streaming capabilities on NVIDIA GPUs. It turns a YAML config (+ optional32OpenAPI spec and custom processors) into a deployable inference pipeline,33packaged as a Docker container or standalone application.3435## Setup3637Inference Builder is part of the DeepStream mono-repo. Resolve it from the38repository root so the skill works from any subdirectory:3940```bash41DEEPSTREAM_ROOT="$(git rev-parse --show-toplevel)"42INFERENCE_BUILDER_ROOT="$DEEPSTREAM_ROOT/tools/inference_builder"43source "$INFERENCE_BUILDER_ROOT/.venv/bin/activate"44python "$INFERENCE_BUILDER_ROOT/builder/main.py" --help45```4647### First-time venv setup4849```bash50cd "$INFERENCE_BUILDER_ROOT"51python3 -m venv .venv52source .venv/bin/activate53pip install -r requirements.txt54```5556### Getting current CLI usage5758Always run with `--help` for authoritative flags — do NOT rely on memorized59syntax:6061```bash62python "$INFERENCE_BUILDER_ROOT/builder/main.py" --help63```6465All paths below are relative to `tools/inference_builder/` unless stated otherwise.6667## Key Directories6869| Path | What lives there |70|------|-----------------|71| `builder/samples/` | Working examples for every backend — read `builder/samples/README.md` for the full index, then start from the closest matching example |72| `schemas/config.schema.json` | Main config schema — all fields, data types, and validation rules |73| `schemas/README.md` | Comprehensive reference: backends, parameters, data types, preprocessors, responders, routing |74| `schemas/backends/` | Per-backend schemas and parameter definitions |75| `templates/backend/` | Jinja templates for each inference backend |76| `templates/responder/` | Jinja templates for server endpoint handlers |77| `lib/` | Shared runtime libraries included in generated pipelines |78| `builder/tests/` | Integration test framework (see `run_tests.sh --help`) |7980## Instructions81821. **Browse samples & schemas** — find the closest match in `builder/samples/`;83 consult `schemas/config.schema.json` and `schemas/backends/` for field reference842. **Write YAML config** — validate against the schema; consult85 `schemas/README.md` for data types, preprocessors, responders, and routing.86 Prefer choices that keep the hot path on GPU and skip avoidable GPU↔CPU87 round-trips; prefer GPU acceleration across the whole pipeline — decode/88 encode, scale & color convert, batch, preprocess, infer, track, and overlay893. **Generate pipeline** — `python builder/main.py <config.yaml> [options]`904. **Prepare model repository** — download models from NGC/HuggingFace into91 the model_repo directory925. **Generate nvinfer config** (DeepStream only) — check if the downloaded93 model already includes runtime config files (nvdsinfer_config.yaml,94 nvdspreprocess_config.yaml); if not, generate them and place alongside95 the model. Skip this step entirely for non-DeepStream backends966. **Select platform & build Docker image** — read `doc/platform-guide.md` to97 choose the correct base image and Dockerfile for the target hardware98 (x86_64, Jetson/Tegra, DGX Spark); then build the image997. **Run & test** — run the container with the model repository mounted;100 for serverless pipelines the container runs to completion; for HTTP101 servers (fastapi/triton/nim) send test requests to verify. If GPU102 resources are available, perform an end-to-end smoke test against the103 running container to validate the full pipeline on real hardware104105## Non-Obvious Knowledge106107- If the user's model request is ambiguous (e.g., unclear model variant,108 version, precision, or source), always ask for clarification before109 proceeding — do not guess.110- Before using any model, gather as much information as possible about it:111 version, architecture, input/output shapes, precision, and expected112 preprocessing. Cross-check this information against the pipeline YAML113 config and any runtime config (e.g., nvdsinfer_config.yaml) to ensure114 consistency — mismatches in input dims, scale factors, or label files115 are common sources of silent failures.116- Config YAML supports runtime env-var substitution: `$ENV_VAR|default_value`117 (auto-typed based on the default).118- Server type `serverless` does NOT need an OpenAPI spec or server definition.119- Custom processor class `name` attribute must exactly match the processor120 name in the YAML config.121- Default to bridge networking plus explicit `-p` port mappings. Recommend122 Docker host-network or host-IPC modes only when the deployment explicitly123 needs host namespace sharing, and call out the security tradeoff before124 opting in.125- Before running containers, check `nvidia-smi` for free GPU memory. If GPUs126 are occupied or low on memory, select a free GPU with `--gpus device=N` or127 ask the user to free resources — OOM failures can be silent or misleading.128- Key runtime env vars: `HTTP_PORT` (default 8000), `LOG_LEVEL` (default INFO),129 `DEBUG` (0/1), `N_CODEC_INSTANCES` (parallel video decoders),130 `MAX_BATCH_SIZE` (live stream frame batching).131- DeepStream backends require a separate `nvdsinfer_config.yaml` in the132 model directory.133- When writing new configs, the required top-level fields are only `name`,134 `model_repo`, and `models` — everything else is optional.135136## Examples137138- "Generate an inference-builder YAML config for a DeepStream detection service139 that reads RTSP, uses Triton, and exposes a FastAPI endpoint."140- "Adapt the dummy sample into a serverless image-inference pipeline and show141 the Docker build and run commands."