qhjqhj00 nepali-summarization-eval
Development of Pre-Trained Transformer-based Models for the Nepali Language — Thapa et al. (2024) (arXiv:2411.15734, 2024)
What this evaluates
Evaluates decoder-based language models on abstractive text summarization for Nepali news articles, testing generation quality and context handling.
Datasets
- Nepali Summarization Dataset (Bhandari 2024) — total 7258; splits: train (5806), test (1452)
Metrics
ROUGE-1— range: percent- Unigram overlap between the generated summary and the reference summary.
ROUGE-2— range: percent- Bigram overlap between the generated summary and the reference summary.
ROUGE-L(primary) — range: percent- Longest common subsequence overlap between the generated summary and the reference summary.
Input / output format
Input: Source Nepali text sequences (typically ≤400 tokens for optimal performance due to model context limits).
Output: Abstractive summary text.
Scoring recipe
def rouge_eval(predictions, golds):
scores = {'ROUGE-1': [], 'ROUGE-2': [], 'ROUGE-L': []}
for pred, gold in zip(predictions, golds):
for metric in scores:
scores[metric].append(calculate_rouge(pred, gold, metric))
return {k: sum(v)/len(v) for k, v in scores.items()}
Common pitfalls
- Model performance degrades significantly on inputs exceeding 400 tokens due to the original 512-token context window limit.
- Evaluation relies on a single publicly available dataset; results may not generalize to other Nepali domains or genres.
Evidence (verbatim from paper)
There were no existing benchmarks for NLG tasks, so we used abstractive summarization for the evaluation of GPT-2... The dataset consists of 7,258 data points, where we used 5,806 (80%) data points for training and the remaining 1,452 (20%) data points for evaluation... We used the ROUGE score (Lin, 2004) Lin and Och ([2004]) as our evaluation metric.
Citation
@misc{thapa2024development,
title={Development of Pre-Trained Transformer-based Models for the Nepali Language},
author={Thapa et al. (2024)},
year={2024},
note={arXiv:2411.15734}
}
- arXiv: 2411.15734