Quantization And Model Compression

Reference-grade guide to shrinking and speeding up LLMs without retraining from scratch — numeric formats (FP8/INT8/INT4), PTQ methods (GPTQ, AWQ, SmoothQuant, bitsandbytes NF4, GGUF k-quants), KV-cache quantization, speculative decoding (Medusa/EAGLE/n-gram), and distillation — with concrete numbers, when each fits, and the quality cliffs.

jpoindexter Updated 2 repo stars

File contents

jpoindexter/design-and-ai-skills commit edc775b055

Frequently asked questions

npx get-skill add jpoindexter/quantization-and-model-compression