# Deployment

> Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says "deploy model", "serve model", "start vLLM server", "launch SGLang", "TRT-LLM deploy", "AutoDeploy", "benchmark throughput", "serve checkpoint", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).

- Skill: `nvidia/deployment` (Agent Skill, multi-file: 10 files)
- Install (CLI): `npx skillmds@latest add nvidia/deployment`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia/deployment/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: Apache-2.0
- Author: NVIDIA (https://skillmd.com/u/nvidia)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nvidia/deployment

---


# Deployment Skill

Serve a model checkpoint as an OpenAI-compatible inference endpoint. Supports vLLM, SGLang, and TRT-LLM (including AutoDeploy).

## Quick Start

Prefer `$SKILL_DIR/scripts/deploy.sh` for standard local deployments — it handles quant detection, health checks, and server lifecycle. Use the raw framework commands in Step 4 when you need flags the script doesn't support, or for remote deployment.

```bash
# Start vLLM server with a ModelOpt checkpoint
"$SKILL_DIR/scripts/deploy.sh" start --model ./qwen3-0.6b-fp8

# Start with SGLang and tensor parallelism
"$SKILL_DIR/scripts/deploy.sh" start --model ./llama-70b-nvfp4 --framework sglang --tp 4

# Start from HuggingFace hub
"$SKILL_DIR/scripts/deploy.sh" start --model nvidia/Llama-3.1-8B-Instruct-FP8

# Test the API
"$SKILL_DIR/scripts/deploy.sh" test

# Check status
"$SKILL_DIR/scripts/deploy.sh" status

# Stop
"$SKILL_DIR/scripts/deploy.sh" stop
```

The script handles: GPU detection, quantization flag auto-detection (FP8 vs FP4), server lifecycle (start/stop/restart/status), health check polling, and API testing.

## Decision Flow

### 0. Check workspace (multi-user / Slack bot)

If `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md`. Before creating a new workspace, check the current session for existing model workspaces — especially if deploying a checkpoint from a prior PTQ run:

```bash
ls "$MODELOPT_WORKSPACE_ROOT/<session_id>/" 2>/dev/null
```

If the user says "deploy the model I just quantized" or references a previous PTQ, find the matching workspace and `cd` into it. The checkpoint should be in that workspace's output directory.

### 1. Identify the checkpoint

Determine what the user wants to deploy:

- **Local quantized checkpoint** (from ptq skill or manual export): look for `hf_quant_config.json` in the directory. If coming from a prior PTQ run in the same workspace, check common output locations: `output/`, `outputs/`, `exported_model/`, or the `--export_path` used in the PTQ command.
- **HuggingFace model hub** (e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`): use directly
- **Unquantized model**: deploy as-is (BF16) or suggest quantizing first with the ptq skill

> **Note:** This skill expects HF-format checkpoints (from PTQ with `--export_fmt hf`). TRT-LLM format checkpoints should be deployed directly with TRT-LLM — see `references/trtllm.md`.

Check the quantization format if applicable:

```bash
cat <checkpoint_path>/hf_quant_config.json 2>/dev/null || echo "No hf_quant_config.json"
```

If not found, also check `config.json` for a `quantization_config` section with `quant_method: "modelopt"`. If neither exists, the checkpoint is unquantized.

### 2. Choose the framework

If the user hasn't specified a framework, recommend based on this priority:

| Situation | Recommended | Why |
|-----------|-------------|-----|
| General use | **vLLM** | Widest ecosystem, easy setup, OpenAI-compatible |
| Best SGLang model support | **SGLang** | Strong DeepSeek/Llama 4 support |
| Maximum optimization | **TRT-LLM** | Best throughput via engine compilation |
| Mixed-precision / AutoQuant | **TRT-LLM AutoDeploy** | Only option for AutoQuant checkpoints |

Check the support matrix in `references/support-matrix.md` to confirm the model + format + framework combination is supported.

### 3. Check the environment

Use the common skill's `environment-setup.md` for GPU detection, local vs remote, and SLURM/Docker/bare metal detection. After completing it you should know: GPU model/count, local or remote, and execution environment.

Then check the **deployment framework** is installed:

```bash
python -c "import vllm; print(f'vLLM {vllm.__version__}')" 2>/dev/null || echo "vLLM not installed"
python -c "import sglang; print(f'SGLang {sglang.__version__}')" 2>/dev/null || echo "SGLang not installed"
python -c "import tensorrt_llm; print(f'TRT-LLM {tensorrt_llm.__version__}')" 2>/dev/null || echo "TRT-LLM not installed"
```

If not installed, consult `references/setup.md`.

**GPU memory estimate** (to determine tensor parallelism):

- BF16: `params × 2 bytes` (8B ≈ 16 GB)
- FP8: `params × 1 byte` (8B ≈ 8 GB)
- FP4: `params × 0.5 bytes` (8B ≈ 4 GB)
- Add ~2-4 GB for KV cache and framework overhead

If the model exceeds single GPU memory, use tensor parallelism (`-tp <num_gpus>`).

### 4. Deploy

Read the framework-specific reference for detailed instructions:

| Framework | Reference file |
|-----------|---------------|
| vLLM | `references/vllm.md` |
| SGLang | `references/sglang.md` |
| TRT-LLM | `references/trtllm.md` |

**Quick-start commands** (for common cases):

#### vLLM

```bash
# Serve as OpenAI-compatible endpoint
python -m vllm.entrypoints.openai.api_server \
    --model <checkpoint_path> \
    --quantization modelopt \
    --tensor-parallel-size <num_gpus> \
    --host 0.0.0.0 --port 8000
```

For NVFP4 checkpoints, use `--quantization modelopt_fp4`.

> **NVFP4 on Blackwell B300/GB300 (sm_103) needs a CUDA-13 image.** From v0.20.0
> on, release tags are CUDA-13 unsuffixed (e.g. `vllm/vllm-openai:v0.26.0`) with
> `-cu129` the CUDA-12 opt-out; v0.19.x and earlier were the other way round
> (`-cu130` = CUDA 13), and `-cu130` no longer exists after v0.20.0. Don't trust
> the tag name — select a tag reporting `CUDA_VERSION` >= 13 in the config blob
> of your platform's child manifest (arm64 Grace/GB300, amd64 x86);
> `TORCH_CUDA_ARCH_LIST` differs between the two. A cu12 build has **no sm_103
> FP4 kernel**, so vLLM loads the checkpoint then dies at engine init with `CUDA
> error: no kernel image is available for execution on the device` (affects the
> `flashinfer` and `cutlass` NVFP4 backends; `marlin` separately fails on
> non-64-divisible layer dims).
> Cross-check via
> `recipes.vllm.ai/<org>/<model>?hardware=b300` (JS-rendered — fetch the raw
> markdown at `github.com/vllm-project/recipes/blob/main/<org>/<model>.md`). For
> multimodal models on sm_103, also pass `--mm-encoder-attn-backend TRITON_ATTN`
> (the default CuTe ViT flash-attn asserts "Only SM 10.x and 11.x").

#### SGLang

```bash
python -m sglang.launch_server \
    --model-path <checkpoint_path> \
    --quantization modelopt \
    --tp <num_gpus> \
    --host 0.0.0.0 --port 8000
```

For NVFP4 checkpoints, use `--quantization modelopt_fp4`.

> **Cross-check SGLang launch flags via the SGLang cookbook** (the SGLang analog
> of `recipes.vllm.ai`): `docs.sglang.io/cookbook/<category>/<org>/<model>` (e.g.
> `.../autoregressive/DeepSeek/DeepSeek-V4`) — authoritative for parallelism, MoE
> backends, strategy flags, Docker image, and min version. Select the variant via
> the URL fragment `#hw=...&variant=...&quant=...&strategy=...&nodes=...`. The
> page is **JS-rendered** — fetch the raw markdown at
> `raw.githubusercontent.com/sgl-project/sglang/main/docs_new/cookbook/<category>/<org>/<model>.mdx`.
> SM120 (RTX PRO 6000) needs the `lmsysorg/sglang:dev` nightly (`:latest` lacks
> SM120). See `references/sglang.md` for the full backend/flag matrix.

#### TRT-LLM (direct)

```python
from tensorrt_llm import LLM, SamplingParams
llm = LLM(model="<checkpoint_path>")
outputs = llm.generate(["Hello, my name is"], SamplingParams(temperature=0.8, top_p=0.95))
```

#### TRT-LLM AutoDeploy

For AutoQuant or mixed-precision checkpoints, see `references/trtllm.md`.

### 5. Verify the deployment

After the server starts, verify it's healthy:

```bash
# Health check
curl -s http://localhost:8000/health

# List models
curl -s http://localhost:8000/v1/models | python -m json.tool

# Test generation
curl -s http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "<model_name>",
        "prompt": "The capital of France is",
        "max_tokens": 32
    }' | python -m json.tool
```

All checks must pass before reporting success to the user.

### 5b. Benchmark throughput/latency (optional)

If the user asks to benchmark, measure throughput/latency, or compare precisions,
use **AIPerf** (Apache-2.0, OpenAI-compatible client benchmark). See
`references/benchmarking.md` for install, the pre-benchmark coherence gate, the
`aiperf profile` flags (notably `--extra-inputs ignore_eos:true`), suggested
token shapes, and how to read `profile_export_aiperf.json`.

### 6. Remote deployment (SSH/SLURM)

If a cluster config exists (`~/.config/modelopt/clusters.yaml`, `.agents/clusters.yaml`, or `.claude/clusters.yaml`), or the user mentions running on a remote machine:

0. **Check container registry auth** — before submitting any SLURM job with a container image, verify credentials exist on the cluster per the common skill's `slurm-setup.md` section 6. If credentials are missing for the image's registry, ask the user to fix auth or switch to an image on an authenticated registry (e.g., NGC). **Do not submit until auth is confirmed.**

1. **Source remote utilities:** Load the common skill, then resolve
   `remote_exec.sh` from that skill's root.

   ```bash
   source "<common-skill-dir>/remote_exec.sh"
   remote_load_cluster
   remote_check_ssh
   remote_detect_env
   ```

2. **Sync the checkpoint** (only if it was produced locally):

   If the checkpoint path is a remote/absolute path (e.g., from a prior PTQ run on the cluster), skip sync — it's already there. Verify with `remote_run "ls <checkpoint_path>/config.json"`. Only sync if the checkpoint is local:

   ```bash
   remote_sync_to <local_checkpoint_path> <session_id>/<model>/checkpoints/
   ```

3. **Deploy based on remote environment:**

   - **SLURM** — see the common skill's `slurm-setup.md` for job script templates (container setup, account/partition discovery). The server command inside the container is the same as Step 4 (e.g., `python -m vllm.entrypoints.openai.api_server --model <path> --quantization modelopt`). After submitting, register the job and set up monitoring per the **monitor skill**. Get the node hostname from `squeue -j $JOBID -o %N`.

   - **Bare metal / Docker** — use `remote_run` to start the server directly:

     ```bash
     remote_run "nohup python -m vllm.entrypoints.openai.api_server --model <path> --port 8000 > deploy.log 2>&1 &"
     ```

4. **Verify remotely:**

   ```bash
   remote_run "curl -s http://localhost:8000/health"
   remote_run "curl -s http://localhost:8000/v1/models"
   ```

5. **Report the endpoint** — include the remote hostname and port so the user can connect (e.g., `http://<node_hostname>:8000`). For SLURM, note that the port is only reachable from within the cluster network.

For NEL-managed deployment (evaluation with self-deployment), use the evaluation skill instead — NEL handles SLURM container deployment, health checks, and teardown automatically.

## Error Handling

| Error | Cause | Fix |
|-------|-------|-----|
| `CUDA error: an illegal memory access` on an **NVFP4 MoE** (`trtllm_fused_moe_dev_kernel.cu`, `deepgemm`) | Fused-MoE FP4 kernel fault on long-context loads; engine dies, all requests 500 | Try `VLLM_USE_FLASHINFER_MOE_FP4=1` + `VLLM_FLASHINFER_MOE_BACKEND=throughput`. **Not always a fix** — on DeepSeek-V4 it moved the fault from TRT-LLM to DeepGEMM. It also makes quantized vs baseline not kernel-matched; record that when reporting deltas. |
| `CUDA out of memory` | Model too large for GPU(s) | Increase `--tensor-parallel-size` or use a smaller model |
| `quantization="modelopt" not recognized` | vLLM/SGLang version too old | Upgrade: vLLM >= 0.10.1, SGLang >= 0.4.10 |
| `hf_quant_config.json not found` | Not a ModelOpt-exported checkpoint | Re-export with `export_hf_checkpoint()`, or remove `--quantization` flag |
| `Connection refused` on health check | Server still starting | Wait 30-60s for large models; check logs for errors |
| `modelopt_fp4 not supported` | Framework doesn't support FP4 for this model | Check support matrix in `references/support-matrix.md` |

## Unsupported Models

If the model is not in the validated support matrix (`references/support-matrix.md`), deployment may fail due to weight key mismatches, missing architecture mappings, or quantized/unquantized layer confusion. Read `references/unsupported-models.md` for the iterative debug loop: **run → read error → diagnose → patch framework source → re-run**. For kernel-level issues, escalate to the framework team rather than attempting fixes.

## Success Criteria

1. Server process is running and healthy (`/health` returns 200)
2. Model is listed at `/v1/models`
3. Test generation produces coherent output
4. Server URL and port are reported to the user
5. If benchmarking was requested, throughput/latency numbers are reported