Nvidia Model Optimizer

NVIDIA Model Optimizer — quantization, pruning, distillation, and speculative decoding for accelerating LLMs, diffusion models, and vision models on NVIDIA GPUs.

mkurman Updated 327 repo stars
Picture from the README

File contents

Overview

NVIDIA Model Optimizer provides state-of-the-art model optimization techniques — quantization (FP8, INT4, NVFP4), pruning, knowledge distillation, and speculative decoding — for models deployable on TensorRT-LLM, SGLang, and vLLM.

Installation

uv pip install nvidia-modelopt

Basic Quantization

import modelopt.torch.quantization as mtq

# FP8 post-training quantization
quant_cfg = mtq.FP8_DEFAULT_CFG
mtq.quantize(model, quant_cfg, forward_loop=calib_loop)

# Export for TensorRT-LLM
from modelopt.torch.export import export_tensorrt_llm_checkpoint
export_tensorrt_llm_checkpoint(model, "model.pt", dtype="fp8")

Pruning + Distillation

import modelopt.torch.pruning as mtp
import modelopt.torch.distill as mtd

# Prune
pruned = mtp.prune(model, ratio=0.3, structure="2:4_sparse")

# Distill
teacher = load_teacher_model()
student = mtd.distill(student_model, teacher, kal="logit", alpha=0.5)

References

mkurman/zorai commit a2c1460bab

Frequently asked questions

npx get-skill add mkurman/nvidia-model-optimizer