Eraser Benchmark Eval

eraser-benchmark-eval

ERASER: A Benchmark to Evaluate Rationalized NLP Models — DeYoung et al. (2019) (arXiv:1911.03429, 2019)

What this evaluates

Evaluates NLP models' ability to generate faithful, task-appropriate rationales for predictions, measuring both alignment with human annotations and causal faithfulness via token perturbation.

Datasets

• Movies — total ?; splits: test (-1)
• FEVER — total ?; splits: test (-1)
• CoS-E — total ?; splits: test (-1)
• eSNLI — total ?; splits: test (-1)

Metrics

• AUPRC (primary) — range: [0, 1]
  • Area Under the Precision-Recall Curve computed by ranking tokens by importance score and comparing against human-annotated rationale spans.
• Comprehensiveness — range: percent
  • Measures the drop in downstream task performance when the top-k% most important tokens are removed, averaged over bins of k.
• Sufficiency — range: percent
  • Measures the drop in downstream task performance when the bottom-k% least important tokens are removed, averaged over bins of k.
• Macro F1 / Accuracy — range: [0, 1]
  • Standard downstream task performance metric used as a baseline to ensure faithfulness metrics do not degrade task capability.

Input / output format

Input: Natural language input (e.g., document, question, or premise-hypothesis pair) with optional human-annotated rationale spans for supervision.

Output: Discrete rationale spans (hard methods) or continuous token-level importance scores (soft methods), plus a final task prediction.

Scoring recipe

scores = model.get_token_importance_scores(input_text)
sorted_indices = argsort(scores, descending=True)
auprc = compute_auc(pr_curve(sorted_indices, human_rationale_mask))
comp_scores, suff_scores = [], []
for k in importance_bins:
    top_k = sorted_indices[:k]
    bottom_k = sorted_indices[-k:]
    perf_drop_comp = downstream_score(input) - downstream_score(remove_tokens(input, top_k))
    perf_drop_suff = downstream_score(input) - downstream_score(remove_tokens(input, bottom_k))
    comp_scores.append(perf_drop_comp)
    suff_scores.append(perf_drop_suff)
comp_avg = mean(comp_scores)
suff_avg = mean(suff_scores)

Common pitfalls

• Confusing rationale agreement metrics (used for hard/discrete selection) with faithfulness metrics (comprehensiveness/sufficiency used for soft/continuous scores).
• Assuming attention weights are faithful explanations; the benchmark shows they often agree with humans but fail perturbation-based faithfulness tests.
• Ignoring variable rationale granularity; sentence-level models cannot be fairly compared to token-level models on datasets requiring short spans.

Evidence (verbatim from paper)

To assess the quality of token scores with respect to human annotations, we report the Area Under the Precision Recall Curve (AUPRC). These scoring functions assign only soft scores to inputs (and may still use all inputs to come to a particular prediction), so we report the metrics intended to measure faithfulness defined above: comprehensiveness and sufficiency, averaged over 'bins' of tokens ordered by importance scores.

Citation

@misc{deyoung2019eraser,
  title={ERASER: A Benchmark to Evaluate Rationalized NLP Models},
  author={DeYoung et al. (2019)},
  year={2019},
  note={arXiv:1911.03429}
}

• arXiv: 1911.03429