Intel Neural Compressor

Intel Neural Compressor — SOTA low-bit LLM quantization (INT8/FP8/INT4/NVFP4), sparsity, pruning, and distillation for PyTorch, TensorFlow, and ONNX Runtime.

mkurman Updated 327 repo stars
Picture from the README

File contents

Overview

Intel Neural Compressor provides low-bit quantization (INT8, FP8, INT4, MXFP4, NVFP4), sparsity, pruning, and knowledge distillation for optimizing models on Intel hardware and beyond.

Installation

uv pip install neural-compressor

Basic Quantization

from neural_compressor import Quantization, config

# Post-training quantization
quantizer = Quantization(config)
q_model = quantizer(model)
q_model.save("quantized_model")

Pruning

from neural_compressor import Pruning

pruner = Pruning(model, config={"pruning_type": "snip_momentum", "target_sparsity": 0.3})
pruned_model = pruner.fit()

References

mkurman/zorai commit 1e5cd17e97

Frequently asked questions

npx get-skill add mkurman/intel-neural-compressor