j4flmao Hardware Acceleration: Extreme Quantization
Mathematics of Sub-8-bit Quantization
Precision reduction targets $W$ (weights) and $A$ (activations) to compress memory footprint and accelerate compute-bound matrix multiplications.
- GPTQ (Post-Training Quantization): Utilizes inverse Hessian approximation to iteratively quantize weights, minimizing the mean squared error (MSE) of layer-wise outputs. Mathematically solves $\min_Q ||WX - QX||"_2^2$ via Cholesky decomposition.
- AWQ (Activation-aware Weight Quantization): Identifies a subset of salient weights based on activation magnitudes ($"|X|"$). Applies optimal scaling to protect salient weights, exploiting the Pareto principle where <1% of weights dominate output variance.
Activation vs Weight Quantization
- Weight-Only (e.g., INT4/INT8): Alleviates memory bandwidth bottlenecks in memory-bound autoregressive decoding.
- Weight-Activation (W8A8): Accelerates compute-bound prefill phases by mapping GEMMs entirely into INT8 Tensor Cores.
FP8 on NVIDIA Hopper Architecture
Hopper introduces FP8 (E4M3 and E5M2 formats).
- E4M3: 4 exponent bits, 3 mantissa bits. Optimal for forward pass (Weights/Activations) due to higher precision.
- E5M2: 5 exponent bits, 2 mantissa bits. Offers expanded dynamic range, crucial for gradient propagation in backward passes. Transformer Engine seamlessly casts FP16/BF16 to FP8, leveraging hardware-native dot-products while mitigating overflow via delayed scaling.
Architecture Flow
%%{init: {"theme": "default", "flowchart": {"useMaxWidth": true}}}%%
flowchart TD
A[FP16/BF16 Model] --> B{Quantization Target}
B -->"|Weights Only| C[Memory-Bound Optimization]
C --> D1[GPTQ: Hessian-based Inverse]
C --> D2[AWQ: Activation-aware Scaling]
D1 --> E[INT4/INT8 Weights]
D2 --> E
B -->|Weights & Activations| F[Compute-Bound Optimization]
F --> G[Hopper FP8 Support]
G --> H1[E4M3 Forward Pass]
G --> H2[E5M2 Backward Pass]
H1 --> I[FP8 Tensor Cores]
H2 --> I