# Qbert Quantization Eval

> Evaluates the impact of ultra-low precision weight quantization (down to 2 bits) on BERTBASE performance across standard NLP tasks. It compares Hessian-guided mixed and group-wise quantization strategies against direct quantization baselines to measure accuracy retention versus model compression. Use when the user wants to benchmark on SST-2, MNLI, CoNLL-03, SQuAD, or asks about evaluating this task. Reports Acc.

- Skill: `qhjqhj00/qbert-quantization-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/qbert-quantization-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/qbert-quantization-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/qbert-quantization-eval

---


# qbert-quantization-eval

> Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT — Shen et al. (2019) (arXiv:1909.05840, 2019)

## What this evaluates

Evaluates the impact of ultra-low precision weight quantization (down to 2 bits) on BERTBASE performance across standard NLP tasks. It compares Hessian-guided mixed and group-wise quantization strategies against direct quantization baselines to measure accuracy retention versus model compression.

## Datasets

- **SST-2** — total ?; splits: dev (-1)
- **MNLI** — total ?; splits: dev (-1)
- **CoNLL-03** — total ?; splits: dev (-1)
- **SQuAD** — total ?; splits: dev (-1)

## Metrics

- `Acc` **(primary)** — range: percent
  - Percentage of correctly predicted class labels out of total instances.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall, computed per class and averaged (macro or micro depending on task convention).
- `Exact Match (EM)` — range: [0, 1]
  - Fraction of predictions where the predicted answer span exactly matches the ground truth span.

## Input / output format

**Input**: Tokenized text sequences (single sentences or sentence pairs) fed into BERTBASE.

**Output**: Predicted class labels for classification tasks (SST-2, MNLI, CoNLL-03) or predicted answer spans for SQuAD.

## Scoring recipe

```python
def compute_metrics(predictions, gold, task):
  if task in ['sst2', 'mnli']:
    return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
  elif task == 'conll03':
    return f1_score(gold, predictions, average='macro')
  elif task == 'squad':
    em = sum(exact_match(p, g) for p, g in zip(predictions, gold)) / len(gold)
    f1 = qa_f1_score(gold, predictions)
    return em, f1
```

## Common pitfalls

- 3-bit quantization is not natively supported in hardware; results often assume casting to 4/8 bits for execution, which affects actual speedup vs memory savings.
- SQuAD fine-tuning exhibits convergence issues (negative Hessian eigenvalues), making it unusually sensitive to quantization compared to other tasks.
- Group-wise quantization accuracy gains saturate at 128 groups; increasing groups further yields diminishing returns while increasing LUT overhead.

## Evidence (verbatim from paper)

> Table 1: Quantization results for BERTBASE on Natural Language Understanding tasks. ... Acc | Size | Size-w/o-e ... Q-BERT | 8 | 8 | 92.88 | 103.9 | 81.2

## Citation

```bibtex
@misc{shen2019qbert,
  title={Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT},
  author={Shen et al. (2019)},
  year={2019},
  note={arXiv:1909.05840}
}
```

- arXiv: 1909.05840