Debatsum Summarization Eval

debatsum-summarization-eval

DebateSum: A large-scale argument mining and summarization dataset — Roush et al. (2020) (arXiv:2011.07251, 2020)

What this evaluates

Evaluates transformer-based models on word-level extractive summarization for policy debate evidence. It measures how well models can identify and extract relevant tokens to form summaries of debate arguments.

Datasets

• DebateSum — total 187386; splits: test (18738); repo https://github.com/Hellisotherpeople/debate2vec

Metrics

• ROUGE F1 (primary) — range: percent
  • Computes the F1 score for unigram (ROUGE-1), bigram (ROUGE-2), and longest common subsequence (ROUGE-L) overlap between the predicted extractive summary and the gold summary. Evaluated using default settings of py-rouge.

Input / output format

Input: A policy debate document represented as a sequence of tokens.

Output: A sequence of token-level labels indicating whether each token should be included in the extractive summary ('underlined' or 'not-underlined').

Scoring recipe

def score(predictions, golds):
    f1_scores = []
    for pred_tokens, gold_tokens in zip(predictions, golds):
        pred_summary = ' '.join([t for t, l in zip(pred_tokens, labels) if l == 'underlined'])
        gold_summary = ' '.join(gold_tokens)
        rouge = ROUGE()
        f1 = rouge.get_scores(pred_summary, gold_summary)[0]['rouge-1']['f']
        f1_scores.append(f1 * 100)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• ROUGE scores are computed on reconstructed summaries from token labels, so tokenization mismatches or boundary errors can disproportionately impact scores.
• The paper uses default py-rouge settings, which may differ from standard NLTK or official ROUGE-1.5.5 implementations, making cross-study comparison difficult.

Evidence (verbatim from paper)

We evaluate our models on a test split of 18,738 documents. The ROUGE metric is used for measuring summarization quality. We evaluate using the default settings of py-rogue on our models. We report the ROUGE F1 scores of these transformer models.

Citation

@misc{roush2020debatesum,
  title={DebateSum: A large-scale argument mining and summarization dataset},
  author={Roush et al. (2020)},
  year={2020},
  note={arXiv:2011.07251}
}

• arXiv: 2011.07251