Glue Lm Eval

glue-lm-eval

Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models — Zhu et al. (2023) (arXiv:2310.13315, 2023)

What this evaluates

Evaluates the quantization performance of pre-trained language models (both discriminative BERT-style and generative GPT-style) across standard NLP classification, regression, and language modeling tasks under data-free zero-shot quantization settings.

Datasets

• GLUE — total ?; splits: dev (-1)
• WikiText2 — total ?; splits: test (-1)
• Penn Treebank (PTB) — total ?; splits: test (-1)
• WikiText103 — total ?; splits: test (-1)

Metrics

• Accuracy (Acc.) — range: [0, 1]
  • Fraction of correctly predicted class labels.
• Pearson correlation (Pear.) — range: [-1, 1]
  • Pearson correlation coefficient between predicted and ground-truth continuous scores.
• Matthew correlation (Mcc.) — range: [-1, 1]
  • Matthews correlation coefficient for binary classification tasks.
• Perplexity (PPL) — range: [0, inf)
  • Exponential of the average negative log-likelihood of the ground-truth tokens.
• GLUE Avg. (primary) — range: [0, 1]
  • Average of task-specific metrics (Acc., Pear., Mcc.) across all GLUE sub-tasks.

Input / output format

Input: Tokenized text sequences for classification/regression tasks; raw text for language modeling tasks.

Output: Class labels or probability distributions for classification; continuous scores for regression; next-token log-probabilities for language modeling.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'classification':
        return np.mean(predictions == golds)
    elif task_type == 'regression':
        return pearsonr(golds, predictions)[0]
    elif task_type == 'binary':
        return matthews_corrcoef(golds, predictions)
    elif task_type == 'lm':
        return np.exp(-np.mean(np.log(predictions + 1e-9)))

Common pitfalls

• Task-specific metrics must be applied correctly (e.g., Mcc. for CoLA, Pear. for STS-B, not Accuracy).
• All reported results are averaged over 5 random seeds to mitigate stochasticity.
• The 'WxAy' notation denotes weight and activation bit-widths respectively, not just weight quantization.
• The method is data-free/zero-shot for quantization; only the teacher model is fine-tuned on task data.

Evidence (verbatim from paper)

For evaluation, we report the performance with Accuracy ("Acc.") metric for most tasks, except the Pearson correlation ("Pear:") for STS-B, the Matthew correlation ("Mcc.") for CoLA, the perplexity (PPL) score for language modeling tasks. We report the averaged results over 5 random seeds to avoid stochasticity.

Citation

@misc{zhu2023zeroshotsharpnessawarequantization,
  title={Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models},
  author={Zhu et al. (2023)},
  year={2023},
  note={arXiv:2310.13315}
}

• arXiv: 2310.13315