Gem Eval

gem-eval

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics — Gehrmann et al. (2021) (arXiv:2102.01672, 2021)

What this evaluates

Evaluates natural language generation models across diverse tasks including content planning, surface realization, and communicative goals. It probes lexical similarity, semantic equivalence, faithfulness, and output diversity using both reference-based and reference-free automated metrics.

Datasets

• CommonGen — total ?; splits: test (-1)
• Czech Restaurant — total ?; splits: test (-1)
• DART — total ?; splits: test (-1)
• E2E clean — total ?; splits: test (-1)
• MLSum — total ?; splits: test (-1)
• Schema-Guided — total ?; splits: test (-1)
• ToTTo — total ?; splits: test (-1)
• XSum — total ?; splits: test (-1)
• WebNLG — total ?; splits: test (-1)
• Turk — total ?; splits: test (-1)
• ASSET — total ?; splits: test (-1)
• WikiLingua — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: [0, 100]
  • Computes modified n-gram precision between the generated text and reference texts, typically with a brevity penalty to discourage overly short outputs.
• ROUGE-1 — range: [0, 100]
  • Measures unigram overlap between generated and reference texts, calculating recall, precision, or F1-score.
• BERTScore — range: [0, 1]
  • Computes cosine similarity between contextualized token embeddings from a pretrained language model (e.g., RoBERTa) for generated and reference texts.
• BLEURT — range: [0, 1]
  • A metric fine-tuned on human ratings using a BERT-based architecture to predict semantic similarity scores.
• QuestEval — range: [0, 1]
  • A QA-based faithfulness metric that generates questions from the reference and checks if the generated summary can answer them correctly.
• MSTTR — range: [0, 1]
  • Mean Segmented Type-Token Ratio: computes the type-token ratio over fixed-length segments (e.g., 100 tokens) and averages them.

Input / output format

Input: Varies by task: natural language text (e.g., source documents for summarization), structured data (e.g., knowledge graph triples for WebNLG/DART), or dialog acts (e.g., E2E).

Output: Natural language generation (e.g., summaries, descriptions, dialog responses) corresponding to the input format.

Scoring recipe

def compute_gem_metrics(predictions, references):
    scores = {}
    for metric_name in ['BLEU', 'ROUGE-1', 'BERTScore', 'BLEURT', 'QuestEval', 'MSTTR']:
        metric_scores = []
        for pred, refs in zip(predictions, references):
            # Use GEM framework or standard libraries to compute metric
            score = get_metric_score(metric_name, pred, refs)
            metric_scores.append(score)
        scores[metric_name] = sum(metric_scores) / len(metric_scores)
    return scores

Common pitfalls

• ROUGE scores can be artificially inflated by increasing the output length of the model.
• Reliability of lexical metrics heavily depends on the quality and quantity of reference translations.
• Diversity metrics (e.g., Distinct-n) often trade off with generation quality.
• QA-based faithfulness metrics like QuestEval do not highly correlate with traditional reference-based metrics.

Evidence (verbatim from paper)

The set of metrics can be computed via the framework described at https://gem-benchmark.com/shared_task which comprises metrics in the following categories: Lexical Similarity (BLEU, ROUGE-1/2/L, METEOR), Semantic Equivalence (BERTScore, BLEURT), Probing for Faithfulness (QuestEval, NUBIA), Diversity (MSTTR, Distinct1/2, H1, H2, Unique1/2), and System Characterization.

Citation

@misc{gohrmann2021gem,
  title={The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics},
  author={Gehrmann et al. (2021)},
  year={2021},
  note={arXiv:2102.01672}
}

• arXiv: 2102.01672