qhjqhj00 source-sentence-detection-eval
Source Identification in Abstractive Summarization — Suhara et al. (2024) (arXiv:2402.04677, 2024)
What this evaluates
Evaluates a model's ability to identify which sentences in a source document contribute to an abstractive summary. It probes source sentence detection capability by ranking candidate sentences based on their inferred relevance to the summary.
Datasets
- SourceSum — total ?; splits: XSum PEGASUS (-1), XSum Ref (-1), CNN/DM PEGASUS (-1), CNN/DM Ref (-1)
Metrics
NDCG(primary) — range: [0, 1]- Normalized Discounted Cumulative Gain. Uses the total number of annotator votes as the relevance score for each sentence, ranking them by predicted contribution.
MAP— range: [0, 1]- Mean Average Precision. Binarizes annotations by assigning a relevance score of 1 if two annotators agree a sentence is a source, and 0 otherwise.
Input / output format
Input: A source document, a summary (generated or reference), and a list of candidate sentences from the document to be ranked.
Output: A ranked list of candidate sentences with associated relevance scores.
Scoring recipe
import math
def compute_ndcg(preds, votes):
dcg = sum(v / math.log2(i + 2) for i, v in enumerate([votes[p] for p in preds]))
idcg = sum(v / math.log2(i + 2) for i, v in enumerate(sorted(votes, reverse=True)))
return dcg / idcg if idcg > 0 else 0.0
def compute_map(preds, gold_binary):
rel_count = 0
prec_sum = 0.0
for i, idx in enumerate(preds):
if gold_binary[idx] == 1:
rel_count += 1
prec_sum += rel_count / (i + 1)
return prec_sum / sum(gold_binary) if sum(gold_binary) > 0 else 0.0
Common pitfalls
- Using threshold-based binary classification instead of ranking metrics introduces arbitrary threshold selection bias.
- MAP requires strict binarization (1 if 2 annotators agree, else 0), whereas NDCG uses continuous vote counts; mixing these conventions invalidates results.
Evidence (verbatim from paper)
To make the evaluation independent of the choice of threshold selection, we used ranking metrics for evaluation, namely NDCG and MAP (Manning et al., 2008). For NDCG, we used the total votes as the score to consider sentences with more votes more important. For MAP calculation, we binarized annotations and considered source sentences if two annotators agree it is relevant.
Citation
@misc{suhara2024sourceidentification,
title={Source Identification in Abstractive Summarization},
author={Suhara et al. (2024)},
year={2024},
note={arXiv:2402.04677}
}
- arXiv: 2402.04677