aselimc Quantization Pipeline
Post-Training Quantization (PTQ)
from torch.quantization import quantize_dynamic
model_int8 = quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)
TensorRT INT8
- Prepare calibration dataset (100-1000 representative samples)
- Choose calibration algorithm: entropy (default), minmax, percentile
- Build engine with INT8 flag
- Validate accuracy: target <1% degradation from FP32
Methods Comparison
| Method | Bits | Needs Training | Best For |
|---|---|---|---|
| PTQ | 8 | No | Quick deployment |
| QAT | 8 | Yes | Max accuracy retention |
| GPTQ | 4 | No (calibration) | LLM serving |
| AWQ | 4 | No (calibration) | LLM serving |
Validation
- Compare accuracy metrics (mAP, perplexity) vs full precision
- Measure latency speedup and memory reduction
- Test on edge cases (out-of-distribution inputs)
Key Libraries
torch.quantization, TensorRT, OpenVINO, ONNX Runtime