Bart Eval

bart-eval

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension — Mike Lewis et al. (arXiv:1910.13461, 2019)

What this evaluates

Evaluates a denoising sequence-to-sequence pre-trained model across discriminative comprehension, abstractive text generation, dialogue response, and machine translation tasks to measure cross-task generalization and generation quality.

Datasets

• SQuAD 1.1 — total ?; splits: test (-1)
• SQuAD 2.0 — total ?; splits: test (-1)
• GLUE — total ?; splits: dev (-1)
• CNN/DailyMail — total ?; splits: test (-1)
• XSum — total ?; splits: test (-1)
• ConvAI2 — total ?; splits: validation (-1)
• ELI5 — total ?; splits: test (-1)
• WMT'16 RO-EN — total ?; splits: test (-1)

Metrics

• ROUGE (primary) — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation. Measures overlap of unigrams, bigrams, and longest common subsequence between generated and reference text.
• BLEU — range: [0, 1]
  • Bilingual Evaluation Understudy. Geometric mean of modified n-gram precisions with a brevity penalty to penalize overly short translations.
• Exact Match (EM) — range: [0, 1]
  • Percentage of predictions that exactly match the ground truth answer span character-for-character.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for token overlap in open-ended QA and dialogue.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances across discriminative tasks.
• Matthews Correlation Coefficient (MCC) — range: [-1, 1]
  • Correlation coefficient between predicted and actual binary classifications, robust to class imbalance.
• Perplexity (PPL) — range: [0, inf)
  • Exponential of the average negative log-likelihood of the reference tokens under the model.

Input / output format

Input: Task-specific text input: document/question/dialogue context for comprehension, source sentence for generation/translation.

Output: Task-specific target: span/label for comprehension, generated summary/response/translation for generation tasks.

Scoring recipe

def score(predictions, gold, task):
    if task in ['summarization', 'qa']:
        return rouge_l(predictions, gold)
    elif task == 'translation':
        return bleu(predictions, gold)
    elif task == 'squad':
        return exact_match(predictions, gold)
    elif task == 'glue':
        return accuracy(predictions, gold)
    elif task == 'convai2':
        return f1(predictions, gold), perplexity(predictions, gold)
    return 0

Common pitfalls

• Generation decoding requires specific beam search settings (size 5, trigram blocking, length penalty tuning) rather than greedy decoding.
• ROUGE and BLEU scores are highly sensitive to tokenization; official tokenizer normalization must be applied for fair comparison.
• SQuAD 2.0 includes unanswerable questions, requiring a threshold-based prediction strategy not needed for SQuAD 1.1.

Evidence (verbatim from paper)

BART outperforms the best previous work, which leverages BERT, by roughly 6.0 points on all ROUGE metrics—representing a significant advance in performance on this problem.

Citation

@misc{lewis2019bart,
  title={BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension},
  author={Mike Lewis et al.},
  year={2019},
  note={arXiv:1910.13461}
}

• arXiv: 1910.13461