Gazeta Russian Summarization Eval

gazeta-russian-summarization-eval

Dataset for Automatic Summarization of Russian News — Gusev (2020) (arXiv:2006.11063, 2020)

What this evaluates

This benchmark evaluates the quality of abstractive and extractive text summarization in Russian. It measures how well generated summaries capture the key information and stylistic qualities of the original news articles compared to human-written references.

Datasets

• Gazeta — total 63435; splits: test (-1); repo https://github.com/IlyaGusev/gazeta

Metrics

• ROUGE (primary) — range: [0, 1]
  • Computes recall and precision of overlapping n-grams (ROUGE-1, ROUGE-2) and longest common subsequence (ROUGE-L).
• BLEU — range: [0, 1]
  • Precision-based metric calculating n-gram overlap between predicted and reference summaries.
• METEOR — range: [0, 1]
  • Weights recall higher than precision and includes synonym/stem matching for n-gram overlap.
• Extraction score — range: [0, 1]
  • Computes normalized lengths of all long non-overlapping common sequences between source text and summary, ensuring the sum is between 0 and 1.
• Plagiarism score — range: [0, 1]
  • Normalized length of the longest common sequence between a text and a summary.

Input / output format

Input: Russian news article text.

Output: Russian summary text.

Scoring recipe

# Tokenize source, prediction, and reference using Razdel tokenizer
src_tokens = razdel.tokenize(source_text)
pred_tokens = razdel.tokenize(prediction_text)
ref_tokens = razdel.tokenize(reference_text)

# Calculate ROUGE-L (longest common subsequence)
rouge_l = len(lcs(src_tokens, pred_tokens)) / len(src_tokens)

# Calculate BLEU (precision-based n-gram overlap)
bleu = compute_bleu(ref_tokens, pred_tokens)

# Calculate METEOR (recall-weighted overlap)
meteor = compute_meteor(ref_tokens, pred_tokens)

# Calculate Extraction/Plagiarism scores
extraction = len(lcs(src_tokens, pred_tokens)) / len(src_tokens)
plagiarism = len(lcs(src_tokens, pred_tokens)) / len(src_tokens)

Common pitfalls

• Only one reference summary is provided per document, which can artificially penalize valid alternative summaries with zero n-gram overlap.
• Early versions of the paper incorrectly reported character-level BLEU scores instead of word-level; researchers must use the provided Razdel tokenizer for consistency.
• Human evaluation did not explicitly instruct annotators to judge abstractiveness, causing a bias toward highly extractive, error-free model outputs over human-written summaries.

Evidence (verbatim from paper)

We measured the quality of summarization with three sets of automatic metrics: ROUGE, BLEU, METEOR. All of them are used in various text generation tasks and are based on the overlaps of N-grams. ROUGE and METEOR are prevalent in text summarization research, and BLEU is a primary automatic metric in machine translation. BLUE is a precision-based metric and does not take recall into account, while ROUGE uses both recall and precision-based metrics in a balanced way, and METEOR weight for the recall part is higher than weight for the precision part. We lower-cased and tokenized reference and predicted summaries with Razdel tokenizer to unify the methodology across all models.

Citation

@misc{gusev2020gazeta,
  title={Dataset for Automatic Summarization of Russian News},
  author={Gusev (2020)},
  year={2020},
  note={arXiv:2006.11063}
}

• arXiv: 2006.11063