Pn Summary Eval

pn-summary-eval

Leveraging ParsBERT and Pretrained mT5 for Persian Abstractive Text Summarization — Farahani et al. (2020) (arXiv:2012.11204, 2020)

What this evaluates

Evaluates the ability of transformer-based models to generate abstractive summaries in Persian. It measures how well generated summaries match reference summaries in terms of lexical overlap and longest common subsequence at the sentence level.

Datasets

• pn-summary — total 93207; splits: train (-1); repo http://github.com/hooshvare/pn-summary

Metrics

• ROUGE-1 F-1 (primary) — range: [0, 1]
  • Computes the F-1 score based on unigram overlap between generated and reference summaries.
• ROUGE-2 F-1 — range: [0, 1]
  • Computes the F-1 score based on bigram overlap between generated and reference summaries.
• ROUGE-L F-1 — range: [0, 1]
  • Computes the F-1 score based on the Longest Common Subsequence (LCS) between generated and reference summaries at the sentence level, ignoring newlines.

Input / output format

Input: Persian news article text (documents crawled from 6 news agency websites)

Output: Abstractive Persian summary generated by the model

Scoring recipe

def compute_rouge_f1(predictions, references, metric_type):
    # metric_type: 'rouge1', 'rouge2', 'rougel'
    # Uses ROUGE package to compute precision, recall, and F-1
    # For ROUGE-L, LCS is computed at sentence level ignoring newlines
    # F-1 = 2 * (precision * recall) / (precision + recall)
    return f1_score

Common pitfalls

• ROUGE metrics only measure lexical overlap and do not capture semantic similarity or factual correctness.
• Token lengths in summaries vary significantly, with most around 30 tokens, which can disproportionately affect F-1 scores if generation length is not controlled.

Evidence (verbatim from paper)

To determine the performance of the models, we use Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric package [28]. This package is widely used for automatic summarization and machine translation evaluation. The metrics included in this package compare an automated summary against a reference summary for each document. There are five different metrics included in this package. We calculate the F-1 score for three of these metrics to show the overall performance of both models on the proposed dataset: - ROUGE-1 (unigram) scoring which computes the overlap of uni-grams between the generated and the reference summaries. - ROUGE-2 (bigram) scoring which computes the overlap of bigrams between the generated and the reference summaries. - ROUGE-L scoring in which the scores are calculated at sentence-level. In this metric new lines are ignored, and Longest Common Subsequence (LCS) is computed between two text pieces.

Citation

@misc{farahani2020parsbert,
  title={Leveraging ParsBERT and Pretrained mT5 for Persian Abstractive Text Summarization},
  author={Farahani et al. (2020)},
  year={2020},
  note={arXiv:2012.11204}
}

• arXiv: 2012.11204