L3cube Mahasum Eval

l3cube-mahasum-eval

L3Cube-MahaSum: A Comprehensive Dataset and BART Models for Abstractive Text Summarization in Marathi — Deshmukh et al. (2024) (arXiv:2410.09184, 2024)

What this evaluates

Evaluates abstractive text summarization models on their ability to generate concise, fluent, and coherent Marathi news summaries from longer source articles. It benchmarks performance against both a newly curated large-scale dataset (MahaSum) and an existing multilingual benchmark (XL-Sum Marathi subset).

Datasets

• XLsum — total 10903; splits: train (8722), test (1090), val (1090); repo https://github.com/csebuetnlp/xl-sum
• MahaSum — total 25374; splits: train (20299), test (2537), val (2537); repo https://github.com/l3cube-pune/MarathiNLP

Metrics

• ROUGE (primary) — range: [0, 1]
  • Recall and precision of overlapping unigrams, bigrams, and skip-bigrams between the generated summary and reference summaries. Typically reported as ROUGE-1, ROUGE-2, and ROUGE-L (longest common subsequence).

Input / output format

Input: Marathi news article text (full document, average ~315-653 words)

Output: Abstractive summary text (average ~18-25 words)

Scoring recipe

def compute_rouge(predictions, references):
    scores = {}
    for rouge_type in ['ROUGE-1', 'ROUGE-2', 'ROUGE-L']:
        r = rouge_score(references, predictions, rouge_types=[rouge_type])
        scores[rouge_type] = r[rouge_type].fmeasure
    return scores

Common pitfalls

• Summaries are highly abstractive and concise, so exact string matching or BLEU will yield artificially low scores; ROUGE is the standard metric used.
• The dataset uses a fixed 80/10/10 train/val/test split; using different splits or cross-validation will change reported performance.
• MahaSum is scraped from specific Marathi news sources (Lokmat, Loksatta), so models may overfit to domain-specific vocabulary and struggle on out-of-domain text.

Evidence (verbatim from paper)

By using MahaSum alongside XL-Sum, we evaluate how well the IndicBART model performs in summarizing Marathi text. ... performance validated via ROUGE metrics.

Citation

@misc{deshmukh2024l3cubemahasum,
  title={L3Cube-MahaSum: A Comprehensive Dataset and BART Models for Abstractive Text Summarization in Marathi},
  author={Deshmukh et al. (2024)},
  year={2024},
  note={arXiv:2410.09184}
}

• arXiv: 2410.09184