Reddit Tifu Summarization Eval

reddit-tifu-summarization-eval

Abstractive Summarization of Reddit Posts with Multi-level Memory Networks — Kim et al. (2018) (arXiv:1811.00783, 2018)

What this evaluates

This evaluation probes a model's ability to generate abstractive summaries from informal, user-generated text and formal documents. It measures how well the model captures long-range dependencies and abstracts key information without relying on extractive heuristics.

Datasets

• Reddit TIFU — total ?; splits: test (-1)
• Newsroom-Abs — total ?; splits: test (-1)
• XSum — total ?; splits: test (-1)

Metrics

• ROUGE-1 (primary) — range: [0, 1]
  • F1 score based on overlapping unigrams between generated and reference summaries. Higher values indicate better performance.
• ROUGE-2 — range: [0, 1]
  • F1 score based on overlapping bigrams between generated and reference summaries. Higher values indicate better performance.
• ROUGE-L — range: [0, 1]
  • F1 score based on the longest common subsequence (LCS) between generated and reference summaries. Higher values indicate better performance.
• Perplexity — range: other
  • Exponential of the average negative log-likelihood of the reference tokens under the model distribution. Lower values indicate better performance.

Input / output format

Input: Source text (Reddit post or article) provided as a sequence of words/tokens.

Output: Abstractive summary generated as a sequence of words/tokens.

Scoring recipe

def evaluate(predictions, gold):
    r1 = rouge_f1(gold, predictions, ngram=1)
    r2 = rouge_f1(gold, predictions, ngram=2)
    rl = rouge_f1(gold, predictions, lcs=True)
    ppl = exp(-mean(log_prob(w, context) for w in gold))
    return {'ROUGE-1': r1, 'ROUGE-2': r2, 'ROUGE-L': rl, 'PPL': ppl}

Common pitfalls

• ROUGE scores may underestimate performance on this dataset because it is highly abstractive and often paraphrases source text rather than copying exact phrases.
• Perplexity is evaluated on the model's internal probability distribution over the reference text, not on the generated output, and lower values indicate better performance (inverse of typical accuracy metrics).
• Human preference evaluation (AMT) uses pairwise comparisons with random ordering; results can be sensitive to prompt phrasing and annotator fatigue.

Evidence (verbatim from paper)

We evaluate the summarization performance with two language metrics: perplexity and standard F1 ROUGE scores Lin (2004). We remind that lower perplexity and higher ROUGE scores indicate better performance.

Citation

@misc{kim2018abstractive,
  title={Abstractive Summarization of Reddit Posts with Multi-level Memory Networks},
  author={Kim et al. (2018)},
  year={2018},
  note={arXiv:1811.00783}
}

• arXiv: 1811.00783