Answersumm Eval

answersumm-eval

AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization — Fabbri et al. (2021) (arXiv:2111.06474, 2021)

What this evaluates

Evaluates multi-perspective answer summarization for community question answering, probing content selection, perspective clustering, abstractive summarization, and factual consistency/coverage.

Datasets

• AnswerSumm — total 4631; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Standard F1 score for binary relevance classification in the SentSelect subtask.
• ROUGE-1/2/L (primary) — range: [0, 1]
  • Standard ROUGE recall/precision/F1 scores measuring n-gram overlap between generated summaries and gold references.
• NLI — range: [0, 1]
  • Natural Language Inference entailment score measuring factual consistency between input and output.
• Semantic Area — range: [0, 1]
  • Metric measuring semantic coverage of the generated summary relative to the source documents.

Input / output format

Input: Question concatenated with source sentences or documents.

Output: Abstractive summary or bullet-point answers.

Scoring recipe

def compute_rouge(predictions, references):
    return rouge1, rouge2, rouge_l

def compute_f1(predictions, references):
    tp = sum(1 for p, r in zip(predictions, references) if p == r == 1)
    fp = sum(1 for p, r in zip(predictions, references) if p == 1 and r == 0)
    fn = sum(1 for p, r in zip(predictions, references) if p == 0 and r == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Low inter-annotator agreement (Fleiss Kappa 0.25) makes the SentSelect subtask highly subjective.
• Models tend to be overly extractive with low novel unigram rates (4%) compared to gold summaries (21%).
• ROUGE scores do not fully align with factual consistency (NLI) or semantic coverage (Semantic Area) metrics.

Evidence (verbatim from paper)

The best results for SentSelect are yielded by RoBERTa relevance classification as illustrated in Table 5. RoBERTa yields an F1 score of 0.49. ... Table 6: ROUGE scores for ClustSumm and Fusion summarization tasks... Table 8: A comparison of model ROUGE, NLI, and Semantic Area scores.

Citation

@misc{fabbri2021answersumm,
  title={AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization},
  author={Fabbri et al. (2021)},
  year={2021},
  note={arXiv:2111.06474}
}

• arXiv: 2111.06474