# L3cube Mahasum Eval

> Evaluates abstractive text summarization models on their ability to generate concise, fluent, and coherent Marathi news summaries from longer source articles. It benchmarks performance against both a newly curated large-scale dataset (MahaSum) and an existing multilingual benchmark (XL-Sum Marathi subset). Use when the user wants to benchmark on XLsum, MahaSum, or asks about evaluating this task. Reports ROUGE.

- Skill: `qhjqhj00/l3cube-mahasum-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/l3cube-mahasum-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/l3cube-mahasum-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/l3cube-mahasum-eval

---


# l3cube-mahasum-eval

> L3Cube-MahaSum: A Comprehensive Dataset and BART Models for Abstractive Text Summarization in Marathi — Deshmukh et al. (2024) (arXiv:2410.09184, 2024)

## What this evaluates

Evaluates abstractive text summarization models on their ability to generate concise, fluent, and coherent Marathi news summaries from longer source articles. It benchmarks performance against both a newly curated large-scale dataset (MahaSum) and an existing multilingual benchmark (XL-Sum Marathi subset).

## Datasets

- **XLsum** — total 10903; splits: train (8722), test (1090), val (1090); repo https://github.com/csebuetnlp/xl-sum
- **MahaSum** — total 25374; splits: train (20299), test (2537), val (2537); repo https://github.com/l3cube-pune/MarathiNLP

## Metrics

- `ROUGE` **(primary)** — range: [0, 1]
  - Recall and precision of overlapping unigrams, bigrams, and skip-bigrams between the generated summary and reference summaries. Typically reported as ROUGE-1, ROUGE-2, and ROUGE-L (longest common subsequence).

## Input / output format

**Input**: Marathi news article text (full document, average ~315-653 words)

**Output**: Abstractive summary text (average ~18-25 words)

## Scoring recipe

```python
def compute_rouge(predictions, references):
    scores = {}
    for rouge_type in ['ROUGE-1', 'ROUGE-2', 'ROUGE-L']:
        r = rouge_score(references, predictions, rouge_types=[rouge_type])
        scores[rouge_type] = r[rouge_type].fmeasure
    return scores
```

## Common pitfalls

- Summaries are highly abstractive and concise, so exact string matching or BLEU will yield artificially low scores; ROUGE is the standard metric used.
- The dataset uses a fixed 80/10/10 train/val/test split; using different splits or cross-validation will change reported performance.
- MahaSum is scraped from specific Marathi news sources (Lokmat, Loksatta), so models may overfit to domain-specific vocabulary and struggle on out-of-domain text.

## Evidence (verbatim from paper)

> By using MahaSum alongside XL-Sum, we evaluate how well the IndicBART model performs in summarizing Marathi text. ... performance validated via ROUGE metrics.

## Citation

```bibtex
@misc{deshmukh2024l3cubemahasum,
  title={L3Cube-MahaSum: A Comprehensive Dataset and BART Models for Abstractive Text Summarization in Marathi},
  author={Deshmukh et al. (2024)},
  year={2024},
  note={arXiv:2410.09184}
}
```

- arXiv: 2410.09184