# Model Quantization

> Model Quantization Mechanics: Precision Reduction and Weight Formatting

- Skill: `j4flmao/model-quantization` (Agent Skill)
- Install (CLI): `npx skillmds@latest add j4flmao/model-quantization`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j4flmao/model-quantization/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: j4flmao (https://skillmd.com/u/j4flmao)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/j4flmao/model-quantization

---

# Model Quantization Mechanics: Precision Reduction and Weight Formatting

## 1. Mathematical Mechanics of Quantization (FP16 to INT4)

Quantization reduces the precision of model weights (and sometimes activations) from 16-bit floating-point (FP16/BF16) to lower bit-widths (e.g., INT4, INT8).
- **Affine Quantization:** Given a tensor of weights $W$, quantization operates via a scaling factor $S$ and a zero-point $Z$.
  $$ W_{quant} = \text{round}\left(\frac{W}{S}\right) + Z $$
  $$ W_{dequant} = S \times (W_{quant} - Z) $$
- **Group-wise Quantization:** Applying a single scale/zero-point across a massive weight matrix leads to severe outlier degradation. Weights are grouped into blocks (e.g., $g=128$), and distinct $S$ and $Z$ are computed per group, mitigating the impact of anomalous activation/weight magnitudes.

## 2. Advanced Quantization Algorithms

### 2.1 GPTQ (Generative Pre-trained Transformer Quantization)
GPTQ is an Optimal Brain Quantization (OBQ) derivative based on approximate second-order Hessian information.
- **Objective:** Minimize the layer-wise reconstruction error $\lVert WX - \hat{W}X \rVert_2^2$.
- **Mechanics:** GPTQ quantizes weights sequentially (column by column). When a weight is quantized, the quantization error is compensated by updating all remaining unquantized weights in the same row. It utilizes a Cholesky decomposition of the inverse Hessian matrix $(H^{-1})$ to compute optimal updates efficiently, enabling the quantization of massive matrices (e.g., 175B parameters) in hours.

### 2.2 AWQ (Activation-aware Weight Quantization)
AWQ preserves performance by avoiding the quantization of "salient" weights (typically ~1% of weights).
- **Salience Identification:** Weights are deemed salient not by their own magnitude, but by the magnitude of their corresponding input activations ($X$).
- **Scale Transformation:** Instead of mixing precision (which is hardware inefficient), AWQ applies a per-channel scaling factor $s$ to multiply the salient weights and divide the corresponding input activations. This artificially reduces the relative quantization error for these critical weights without altering the mathematical output of the layer.

## 3. Storage and Execution Formats: GGUF

GGUF (GPT-Generated Unified Format) supersedes GGML as the standard for CPU/CPU+GPU hybrid inference.
- **Memory Mapping (mmap):** GGUF is designed for direct memory-mapping. The entire file (metadata + tensor data) can be mapped into RAM without parsing overhead.
- **Tensor Layout:** GGUF stores tensors with varying quantization schemes (e.g., `Q4_K_M`, where K denotes k-quants). Tensors are block-quantized, often utilizing a multi-level quantization approach (e.g., super-blocks containing scales, which are themselves quantized).
- **Extensibility:** Utilizes a robust key-value metadata store allowing arbitrary hyperparameter injection (RoPE scaling factors, EOS token IDs) preventing the need for external configuration files.

## 4. Quantization Topology

```mermaid
%%{init: {"theme": "default", "flowchart": {"useMaxWidth": true}}}%%
flowchart TD
    A[FP16/BF16 Model] --> B{Quantization Algorithm}
    
    subgraph DataDependentCalibrationDataDependentCalibration ["Data-Dependent Calibration<br><br><br>"]
        B -->|Hessian Matrix Compensation| C[GPTQ]
        B -->|Activation Salience Scaling| D[AWQ]
        C --> E[Column-wise Cholesky Update]
        D --> F[Identify High-Magnitude Activations]
        F --> G[Per-channel Scale Factor s]
    end
    
    subgraph WeightCompressionWeightCompression ["Weight Compression<br><br><br>"]
        E --> H[INT4 Weights + FP16 Scales/Zero-Points]
        G --> H
    end
    
    H --> I{Serialization Format}
    I -->|mmap Optimization| J[GGUF Format]
    I -->|GPU Native| K[Safetensors / AWQ Format]
    
    J --> L[CPU/Metal Execution via llama.cpp]
    K --> M[CUDA Execution via ExLlamaV2/vLLM]
```