Qbert Quantization Eval

qbert-quantization-eval

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT — Shen et al. (2019) (arXiv:1909.05840, 2019)

What this evaluates

Evaluates the impact of ultra-low precision weight quantization (down to 2 bits) on BERTBASE performance across standard NLP tasks. It compares Hessian-guided mixed and group-wise quantization strategies against direct quantization baselines to measure accuracy retention versus model compression.

Datasets

• SST-2 — total ?; splits: dev (-1)
• MNLI — total ?; splits: dev (-1)
• CoNLL-03 — total ?; splits: dev (-1)
• SQuAD — total ?; splits: dev (-1)

Metrics

• Acc (primary) — range: percent
  • Percentage of correctly predicted class labels out of total instances.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall, computed per class and averaged (macro or micro depending on task convention).
• Exact Match (EM) — range: [0, 1]
  • Fraction of predictions where the predicted answer span exactly matches the ground truth span.

Input / output format

Input: Tokenized text sequences (single sentences or sentence pairs) fed into BERTBASE.

Output: Predicted class labels for classification tasks (SST-2, MNLI, CoNLL-03) or predicted answer spans for SQuAD.

Scoring recipe

def compute_metrics(predictions, gold, task):
  if task in ['sst2', 'mnli']:
    return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
  elif task == 'conll03':
    return f1_score(gold, predictions, average='macro')
  elif task == 'squad':
    em = sum(exact_match(p, g) for p, g in zip(predictions, gold)) / len(gold)
    f1 = qa_f1_score(gold, predictions)
    return em, f1

Common pitfalls

• 3-bit quantization is not natively supported in hardware; results often assume casting to 4/8 bits for execution, which affects actual speedup vs memory savings.
• SQuAD fine-tuning exhibits convergence issues (negative Hessian eigenvalues), making it unusually sensitive to quantization compared to other tasks.
• Group-wise quantization accuracy gains saturate at 128 groups; increasing groups further yields diminishing returns while increasing LUT overhead.

Evidence (verbatim from paper)

Table 1: Quantization results for BERTBASE on Natural Language Understanding tasks. ... Acc | Size | Size-w/o-e ... Q-BERT | 8 | 8 | 92.88 | 103.9 | 81.2

Citation

@misc{shen2019qbert,
  title={Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT},
  author={Shen et al. (2019)},
  year={2019},
  note={arXiv:1909.05840}
}

• arXiv: 1909.05840