ML Inference Optimization
Overview
Use this skill for reducing inference latency, increasing throughput, shrinking memory, lowering cost, and deploying optimized models safely. Optimization must be benchmark-driven: define workload, input shapes, concurrency, SLOs, hardware, runtime, numerical tolerance, and quality metrics before changing the model.
Optimization Workflow
- Establish a baseline with realistic data, preprocessing, postprocessing, network overhead, warmup, and concurrency.
- Profile bottlenecks: CPU preprocessing, model compute, memory bandwidth, GPU utilization, serialization, queueing, retrieval, or downstream calls.
- Apply the least risky optimization first: batching, compilation, precision, runtime tuning, then compression.
- Revalidate accuracy, calibration, fairness slices, robustness, and numerical stability after every change.
- Measure p50, p95, p99 latency, throughput, memory, cold start, error rate, and cost per 1,000 predictions.
Export and Compilation
Export paths should be chosen by deployment target. Compare exported output with framework output on representative inputs. Define tolerances per output type. Include preprocessing and postprocessing in the benchmark; many failures occur outside the core model.
PyTorch to ONNX Export Blueprint
import torch
import torch.nn as nn
def export_to_onnx(model: nn.Module, dummy_input: torch.Tensor, onnx_path: str):
model.eval()
# Define input and output names, and dynamic axes for batch size and sequence length
input_names = ["input_ids"]
output_names = ["logits"]
dynamic_axes = {
"input_ids": {0: "batch_size", 1: "seq_len"},
"logits": {0: "batch_size"}
}
torch.onnx.export(
model,
dummy_input,
onnx_path,
export_params=True,
opset_version=17, # Modern stable opset version
do_constant_folding=True,
input_names=input_names,
output_names=output_names,
dynamic_axes=dynamic_axes
)
PyTorch torch.compile Optimization Setup
torch.compile leverages the Inductor compiler for massive speedups on modern GPUs.
import torch
def compile_model(model: torch.nn.Module, mode: str = "default"):
# Modes:
# - "default": Balance compile time and inference speed
# - "reduce-overhead": Reduce framework overhead (uses CUDA graphs, good for small batch sizes)
# - "max-autotune": Perform exhaustive kernel tuning (longest compile time, fastest runtime)
optimized_model = torch.compile(
model,
mode=mode,
fullgraph=False,
dynamic=True # Handles variable sequence lengths/batch sizes efficiently
)
return optimized_model
Batching and Concurrency
Dynamic batching is often the biggest serving win for GPU endpoints. Tune max batch size and queue delay together: larger batches improve throughput but can harm tail latency. For LLMs, distinguish prefill and decode phases; sequence length, KV cache, and concurrent requests dominate memory. Use continuous batching or specialized LLM serving runtimes when appropriate.
For CPU services, concurrency may improve utilization until memory bandwidth, GIL/runtime constraints, or thread oversubscription dominate. Pin thread counts for BLAS/OpenMP/ONNX Runtime and benchmark rather than accepting defaults.
Triton Inference Server Configuration
Triton is the industry standard for high-performance multi-framework serving.
Triton config.pbtxt Template
name: "text_classifier"
platform: "onnxruntime_onnx"
max_batch_size: 32
input [
{
name: "input_ids"
data_type: TYPE_INT64
dims: [ -1 ] # Dynamic sequence length
}
]
output [
{
name: "logits"
data_type: TYPE_FP32
dims: [ 5 ] # Number of classes
}
]
# Enable Dynamic Batching
dynamic_batching {
max_queue_delay_microseconds: 5000 # Wait up to 5ms for concurrent requests
preferred_batch_size: [ 8, 16, 32 ]
}
# GPU Instance Groups
instance_group [
{
count: 2 # Host 2 model execution instances per GPU
kind: KIND_GPU
}
]
Precision and Quantization
| Method |
Best for |
Caveats |
| fp16/bf16 |
GPU/accelerator throughput and memory reduction |
Validate numerically sensitive ops |
| INT8 post-training quantization |
CNNs, transformers, tabular neural nets with calibration data |
Calibration set must represent production |
| Quantization-aware training |
Accuracy-sensitive low precision deployments |
Adds training complexity |
| INT4/GPTQ/AWQ/bitsandbytes |
Large language model memory reduction |
Can degrade reasoning, rare tokens, calibration, tool use |
| Weight-only quantization |
LLM serving where activations remain higher precision |
Speedup depends on kernels/hardware |
Quantization changes model behavior. Re-run task metrics, calibration, slice metrics, and safety checks. For RAG or embedding models, verify retrieval recall and embedding similarity distribution after quantization.
Pruning, Sparsity, and Distillation
Pruning removes weights, channels, heads, or layers. Unstructured sparsity may not speed inference without hardware/runtime support; structured pruning is more likely to reduce latency. Knowledge distillation trains a smaller student to mimic a larger teacher.
TensorRT Compilation Recipes (trtexec)
NVIDIA TensorRT compiled engines deliver maximum throughput on GPUs.
1. Build FP16 Engine with Dynamic Shapes
trtexec --onnx=model.onnx \
--saveEngine=model.engine \
--fp16 \
--minShapes=input_ids:1x1 \
--optShapes=input_ids:16x128 \
--maxShapes=input_ids:32x512
2. Build INT8 Engine with Calibration Data
trtexec --onnx=model.onnx \
--saveEngine=model_int8.engine \
--int8 \
--calib=calibration_profile.cache \
--minShapes=input_ids:1x1 \
--optShapes=input_ids:16x128 \
--maxShapes=input_ids:32x512
Serving Runtimes
| Runtime |
Use when |
| Triton Inference Server |
Multi-framework GPU/CPU serving, dynamic batching, ensembles, model repository, high throughput |
| TorchServe |
PyTorch model serving with handlers and management APIs |
| TensorFlow Serving |
TensorFlow SavedModel serving and versioned model management |
| ONNX Runtime Server/custom API |
Portable ONNX inference with lightweight service control |
| BentoML |
Packaging models as APIs, deployment flexibility, Python-centric workflows |
| Seldon/KServe |
Kubernetes-native model serving, canaries, inference graphs, enterprise ops |
| vLLM / TGI |
Continuous batching, PagedAttention, KV cache management, tensor parallel LLM serving |
Use model repository versioning and readiness checks. Separate health checks from expensive inference. For GPU serving, avoid loading multiple memory-heavy models per device unless capacity is proven.
Memory Optimization
For transformers, memory is driven by weights plus KV cache. Reduce memory with quantization, tensor parallelism, smaller max sequence length, paged attention, cache eviction, smaller batch/concurrency, or distilled models. For diffusion and vision models, use attention slicing, VAE tiling, fp16/bf16, memory-efficient attention, and model offload where latency allows.
Edge Deployment
Edge constraints include binary size, RAM, thermal throttling, offline operation, intermittent updates, privacy, and hardware acceleration. Prefer TFLite/Core ML/OpenVINO/ONNX depending on target. Test on real devices.
Benchmarking Pitfalls
- Measuring only warm single-request latency while production has concurrent traffic.
- Ignoring tokenization, image decode, retrieval, serialization, or network time.
- Reporting average latency instead of p95/p99.
- Comparing quantized and baseline models on different data.
- Building TensorRT engines for one shape then serving unsupported dynamic shapes.
- Increasing batch size until throughput improves but SLO fails.
Sources
1---
2name: ml-inference-optimization
3description: |
4 This skill should be used when the user asks to optimize, export, serve, compress, or accelerate ML inference. PROACTIVELY activate for: (1) latency, throughput, p95/p99, batching, concurrency, KV cache, memory, or cost issues, (2) quantization INT8/INT4, GPTQ, AWQ, bitsandbytes, pruning, sparsity, distillation, (3) ONNX export, ONNX Runtime, TensorRT, TorchScript, torch.compile, XLA, OpenVINO, Core ML, TFLite, (4) Triton, TorchServe, TF Serving, BentoML, Seldon, KServe configuration, (5) edge deployment, CPU/GPU/TPU/Inferentia serving. Provides: hardware-aware inference optimization and safe benchmarking.
5metadata:
6 source: https://skillmd.com/skills/josiahsiegel/ml-inference-optimization
7---
8
9# ML Inference Optimization
10
11## Overview
12
13Use this skill for reducing inference latency, increasing throughput, shrinking memory, lowering cost, and deploying optimized models safely. Optimization must be benchmark-driven: define workload, input shapes, concurrency, SLOs, hardware, runtime, numerical tolerance, and quality metrics before changing the model.
14
15## Optimization Workflow
16
171. Establish a baseline with realistic data, preprocessing, postprocessing, network overhead, warmup, and concurrency.
182. Profile bottlenecks: CPU preprocessing, model compute, memory bandwidth, GPU utilization, serialization, queueing, retrieval, or downstream calls.
193. Apply the least risky optimization first: batching, compilation, precision, runtime tuning, then compression.
204. Revalidate accuracy, calibration, fairness slices, robustness, and numerical stability after every change.
215. Measure p50, p95, p99 latency, throughput, memory, cold start, error rate, and cost per 1,000 predictions.
22
23## Export and Compilation
24
25Export paths should be chosen by deployment target. Compare exported output with framework output on representative inputs. Define tolerances per output type. Include preprocessing and postprocessing in the benchmark; many failures occur outside the core model.
26
27### PyTorch to ONNX Export Blueprint
28```python
29import torch
30import torch.nn as nn
31
32def export_to_onnx(model: nn.Module, dummy_input: torch.Tensor, onnx_path: str):
33 model.eval()
34
35 # Define input and output names, and dynamic axes for batch size and sequence length
36 input_names = ["input_ids"]
37 output_names = ["logits"]
38 dynamic_axes = {
39 "input_ids": {0: "batch_size", 1: "seq_len"},
40 "logits": {0: "batch_size"}
41 }
42
43 torch.onnx.export(
44 model,
45 dummy_input,
46 onnx_path,
47 export_params=True,
48 opset_version=17, # Modern stable opset version
49 do_constant_folding=True,
50 input_names=input_names,
51 output_names=output_names,
52 dynamic_axes=dynamic_axes
53 )
54```
55
56### PyTorch `torch.compile` Optimization Setup
57`torch.compile` leverages the Inductor compiler for massive speedups on modern GPUs.
58```python
59import torch
60
61def compile_model(model: torch.nn.Module, mode: str = "default"):
62 # Modes:
63 # - "default": Balance compile time and inference speed
64 # - "reduce-overhead": Reduce framework overhead (uses CUDA graphs, good for small batch sizes)
65 # - "max-autotune": Perform exhaustive kernel tuning (longest compile time, fastest runtime)
66
67 optimized_model = torch.compile(
68 model,
69 mode=mode,
70 fullgraph=False,
71 dynamic=True # Handles variable sequence lengths/batch sizes efficiently
72 )
73 return optimized_model
74```
75
76## Batching and Concurrency
77
78Dynamic batching is often the biggest serving win for GPU endpoints. Tune max batch size and queue delay together: larger batches improve throughput but can harm tail latency. For LLMs, distinguish prefill and decode phases; sequence length, KV cache, and concurrent requests dominate memory. Use continuous batching or specialized LLM serving runtimes when appropriate.
79
80For CPU services, concurrency may improve utilization until memory bandwidth, GIL/runtime constraints, or thread oversubscription dominate. Pin thread counts for BLAS/OpenMP/ONNX Runtime and benchmark rather than accepting defaults.
81
82## Triton Inference Server Configuration
83
84Triton is the industry standard for high-performance multi-framework serving.
85
86### Triton `config.pbtxt` Template
87```protobuf
88name: "text_classifier"
89platform: "onnxruntime_onnx"
90max_batch_size: 32
91
92input [
93 {
94 name: "input_ids"
95 data_type: TYPE_INT64
96 dims: [ -1 ] # Dynamic sequence length
97 }
98]
99
100output [
101 {
102 name: "logits"
103 data_type: TYPE_FP32
104 dims: [ 5 ] # Number of classes
105 }
106]
107
108# Enable Dynamic Batching
109dynamic_batching {
110 max_queue_delay_microseconds: 5000 # Wait up to 5ms for concurrent requests
111 preferred_batch_size: [ 8, 16, 32 ]
112}
113
114# GPU Instance Groups
115instance_group [
116 {
117 count: 2 # Host 2 model execution instances per GPU
118 kind: KIND_GPU
119 }
120]
121```
122
123## Precision and Quantization
124
125| Method | Best for | Caveats |
126|---|---|---|
127| fp16/bf16 | GPU/accelerator throughput and memory reduction | Validate numerically sensitive ops |
128| INT8 post-training quantization | CNNs, transformers, tabular neural nets with calibration data | Calibration set must represent production |
129| Quantization-aware training | Accuracy-sensitive low precision deployments | Adds training complexity |
130| INT4/GPTQ/AWQ/bitsandbytes | Large language model memory reduction | Can degrade reasoning, rare tokens, calibration, tool use |
131| Weight-only quantization | LLM serving where activations remain higher precision | Speedup depends on kernels/hardware |
132
133Quantization changes model behavior. Re-run task metrics, calibration, slice metrics, and safety checks. For RAG or embedding models, verify retrieval recall and embedding similarity distribution after quantization.
134
135## Pruning, Sparsity, and Distillation
136
137Pruning removes weights, channels, heads, or layers. Unstructured sparsity may not speed inference without hardware/runtime support; structured pruning is more likely to reduce latency. Knowledge distillation trains a smaller student to mimic a larger teacher.
138
139## TensorRT Compilation Recipes (trtexec)
140
141NVIDIA TensorRT compiled engines deliver maximum throughput on GPUs.
142
143### 1. Build FP16 Engine with Dynamic Shapes
144```bash
145trtexec --onnx=model.onnx \
146 --saveEngine=model.engine \
147 --fp16 \
148 --minShapes=input_ids:1x1 \
149 --optShapes=input_ids:16x128 \
150 --maxShapes=input_ids:32x512
151```
152
153### 2. Build INT8 Engine with Calibration Data
154```bash
155trtexec --onnx=model.onnx \
156 --saveEngine=model_int8.engine \
157 --int8 \
158 --calib=calibration_profile.cache \
159 --minShapes=input_ids:1x1 \
160 --optShapes=input_ids:16x128 \
161 --maxShapes=input_ids:32x512
162```
163
164## Serving Runtimes
165
166| Runtime | Use when |
167|---|---|
168| Triton Inference Server | Multi-framework GPU/CPU serving, dynamic batching, ensembles, model repository, high throughput |
169| TorchServe | PyTorch model serving with handlers and management APIs |
170| TensorFlow Serving | TensorFlow SavedModel serving and versioned model management |
171| ONNX Runtime Server/custom API | Portable ONNX inference with lightweight service control |
172| BentoML | Packaging models as APIs, deployment flexibility, Python-centric workflows |
173| Seldon/KServe | Kubernetes-native model serving, canaries, inference graphs, enterprise ops |
174| vLLM / TGI | Continuous batching, PagedAttention, KV cache management, tensor parallel LLM serving |
175
176Use model repository versioning and readiness checks. Separate health checks from expensive inference. For GPU serving, avoid loading multiple memory-heavy models per device unless capacity is proven.
177
178## Memory Optimization
179
180For transformers, memory is driven by weights plus KV cache. Reduce memory with quantization, tensor parallelism, smaller max sequence length, paged attention, cache eviction, smaller batch/concurrency, or distilled models. For diffusion and vision models, use attention slicing, VAE tiling, fp16/bf16, memory-efficient attention, and model offload where latency allows.
181
182## Edge Deployment
183
184Edge constraints include binary size, RAM, thermal throttling, offline operation, intermittent updates, privacy, and hardware acceleration. Prefer TFLite/Core ML/OpenVINO/ONNX depending on target. Test on real devices.
185
186## Benchmarking Pitfalls
187
188- Measuring only warm single-request latency while production has concurrent traffic.
189- Ignoring tokenization, image decode, retrieval, serialization, or network time.
190- Reporting average latency instead of p95/p99.
191- Comparing quantized and baseline models on different data.
192- Building TensorRT engines for one shape then serving unsupported dynamic shapes.
193- Increasing batch size until throughput improves but SLO fails.
194
195## Sources
196
197- NVIDIA Triton Inference Server documentation: https://docs.nvidia.com/deeplearning/triton-inference-server/
198- NVIDIA TensorRT documentation: https://docs.nvidia.com/deeplearning/tensorrt/
199- ONNX Runtime documentation: https://onnxruntime.ai/docs/
200- PyTorch export and compilation docs: https://pytorch.org/docs/stable/
201- TensorFlow Lite documentation: https://www.tensorflow.org/lite
202- BentoML documentation: https://docs.bentoml.com/