Zeroquant Eval

zeroquant-eval

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers — Yao et al. (2022) (arXiv:2206.01861, 2022)

What this evaluates

Evaluates the accuracy and inference latency of post-training quantized Transformer models (BERT and GPT-3-style) on standard NLP benchmarks and language modeling tasks.

Datasets

• GLUE benchmark — total ?; splits: dev (-1); HF glue
• 20 zero-shot evaluation tasks — total ?; splits: test (-1)
• PTB / Wikitext-2 / Wikitext-103 — total ?; splits: test (-1)

Metrics

• average accuracy (primary) — range: percent
  • Mean of task-specific classification or QA accuracies across the benchmark tasks.
• perplexity (PPL) — range: other
  • Exponential of the negative average log-likelihood of the ground-truth tokens in the language modeling tasks.
• latency — range: other
  • End-to-end average time in milliseconds to generate the first 50 tokens or process the entire model.

Input / output format

Input: Text prompts, sentences, or passages for classification, question answering, or language modeling tasks.

Output: Predicted class labels for classification/QA tasks, or generated token sequences for language modeling.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_ppl(logits, gold):
    log_probs = -F.cross_entropy(logits, gold, reduction='none')
    return torch.exp(log_probs.mean())

Common pitfalls

• The notation WxAy denotes x-bit weight quantization and y-bit activation quantization, with mixed precision (e.g., W4/8) applying different bit-widths to different layers (e.g., MHSA vs FFC).
• Latency speedups are highly dependent on batch size and sequence length; results vary significantly across configurations.
• LKD (Layer-by-Layer Knowledge Distillation) requires a small calibration set and GPU time (seconds to hours), contradicting a strict 'zero-cost' assumption for aggressive quantization.

Evidence (verbatim from paper)

Comparing ZeroQuant with PTQ using W8A8, we can reduce the accuracy gap from 1.1% to 0.2% and the perplexity (PPL) gap from 4.7 to 0.2 with no activation range calibration cost.

Citation

@misc{yao2022zeroquant,
  title={ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers},
  author={Yao et al. (2022)},
  year={2022},
  note={arXiv:2206.01861}
}

• arXiv: 2206.01861