qhjqhj00 varta-headline-eval
V=arta: A Large-Scale Headline-Generation Dataset for Indic Languages — Aralikatte et al. (2023) (arXiv:2305.05858, 2023)
What this evaluates
Evaluates abstractive headline generation across 15 Indic languages and English. It probes cross-lingual transfer, script normalization effects, and the impact of language-family-specific pretraining on low-resource generation.
Datasets
- Varta — total ?; splits: train (-1), test (-1); repo https://github.com/rahular/varta
Metrics
ROUGE-L(primary) — range: percent- Longest Common Subsequence (LCS) recall and precision between generated and reference headlines, combined into a single F-measure score.
Input / output format
Input: Article text (document) in one of 15 Indic languages or English.
Output: Single-line generated headline string.
Scoring recipe
def rouge_l(predictions, references):
lcs_scores = []
for pred, ref in zip(predictions, references):
lcs = longest_common_subsequence(pred.split(), ref.split())
lcs_scores.append(2 * len(lcs) / (len(pred.split()) + len(ref.split())))
return sum(lcs_scores) / len(lcs_scores) * 100
Common pitfalls
- Evaluating on transliterated scripts (Latin/Devanagari) vs original scripts changes tokenization and can inflate/deflate scores.
- Cross-lingual zero-shot evaluation requires strict language-code matching to avoid contamination.
- ROUGE-L is a surface-form metric that may penalize valid paraphrases or stylistic variations in headlines.
Evidence (verbatim from paper)
We finetune each model described in §4.2 on Värta in five settings: (i) en: finetune only on English data from the SMALL training set, and evaluate on all language test sets in original scripts. ... Table 3: Headline generation results for the three baseline models trained in all five data settings: English only (en), Hindi only (hi), Latin transliterated data (latin), Devanagari transliterated data (dvn.), and original script data (all). Only ROUGE-L scores are shown here.
Citation
@misc{aralikatte2023varta,
title={V\=arta: A Large-Scale Headline-Generation Dataset for Indic Languages},
author={Aralikatte et al. (2023)},
year={2023},
note={arXiv:2305.05858}
}
- arXiv: 2305.05858