Bloom Empirical Eval

bloom-empirical-eval

Understanding BLOOM: An empirical study on diverse NLP tasks — Dakle et al. (2022) (arXiv:2211.14865, 2022)

What this evaluates

Evaluates BLOOM model variants against BERT-style and GPT-style baselines across diverse NLP tasks including text classification, question answering, zero/few-shot learning, multilingual transfer, and text generation.

Datasets

• GLUE — total ?; splits: validation (-1)
• SQuAD — total ?; splits: train (-1), validation (-1)
• XNLI — total ?; splits: validation (-1)
• MARC — total ?; splits: test (-1)
• Zero/FSL Benchmarks — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels out of total samples. Used for GLUE, MARC, XNLI, and FSL tasks.
• exact-match — range: [0, 1]
  • Fraction of predictions where the generated answer exactly matches the ground truth span. Used for SQuAD.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall over token overlap between predicted and gold answers. Used for SQuAD.
• BLEU — range: [0, 1]
  • Geometric mean of modified n-gram precisions with brevity penalty. Used for text generation evaluation.
• Matthews correlation — range: other
  • Correlation coefficient between observed and predicted binary classifications. Used for COLA.

Input / output format

Input: Varies by task: fine-tuning inputs for GLUE/SQuAD/MARC/XNLI; prompt-based inputs with 0/1/4/8 examples for FSL tasks; context+question formatted as '[CLS]context[SEP]question[SEP]' for SQuAD; prompts for toxicity generation.

Output: Task-specific: class labels for classification, span indices for QA, generated text for TG/toxicity.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task == 'classification':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task == 'qa':
        em = sum(p == g for p, g in zip(predictions, golds)) / len(golds)
        # F1 computed via token overlap
        return em, f1_score(golds, predictions)
    elif task == 'generation':
        return bleu_score(golds, predictions)
    elif task == 'cola':
        return matthews_corrcoef(golds, predictions)

Common pitfalls

• MNLI scores are averaged over matched and mismatched subtasks, not reported separately.
• BLOOM models initially predict a single class for COLA/SST2 due to prompt/tokenization issues, requiring padding strategy changes to fix.
• SQuAD uses '[CLS]' token not in BLOOM's vocabulary, affecting no-answer questions; replacing it with '' improves performance.

Evidence (verbatim from paper)

For the MNLI task, we report the average of the matched and mismatched accuracy scores. ... The table shows that the bloom-560m model outperforms gpt2-medium on the SQUAD v.1.1 dataset. Although the metric scores are significantly lower compared to BERT (~73% exact match), the results show that when compared to gpt2-medium, a similar architecture style model, bloom-560m performs better.

Citation

@misc{dakle2022understandingbloom,
  title={Understanding BLOOM: An empirical study on diverse NLP tasks},
  author={Dakle et al. (2022)},
  year={2022},
  note={arXiv:2211.14865}
}

• arXiv: 2211.14865