Liputan6 Eval

liputan6-eval

Liputan6: A Large-scale Indonesian Dataset for Text Summarization — Koto et al. (2020) (arXiv:2011.00679, 2020)

What this evaluates

Evaluates the ability of models to generate concise, accurate summaries of Indonesian news articles. It probes both extractive and abstractive summarization capabilities, highlighting performance gaps on more abstract summaries and the limitations of n-gram overlap metrics.

Datasets

• Liputan6 — total ?; splits: train (-1), dev (-1), canonical test (-1), Xtreme test (-1); repo https://github.com/fajri91/sum_liputan6

Metrics

• ROUGE F-1 (R1, R2, RL) (primary) — range: [0, 1]
  • Computes F-1 scores based on unigram (R1), bigram (R2), and longest common subsequence (RL) overlap between the generated summary and the reference summary.
• BERTSCORE (F-1) — range: [0, 1]
  • Computes F-1 scores based on contextual embeddings from bert-base-multilingual-cased (layer 9), matching the protocol used for machine translation evaluation.

Input / output format

Input: Indonesian news article (document text)

Output: Generated summary text

Scoring recipe

def score(predictions, references):
    scores = {'R1': 0, 'R2': 0, 'RL': 0, 'BERTSCORE': 0}
    for pred, ref in zip(predictions, references):
        scores['R1'] += rouge_f1(pred, ref, ngram=1)
        scores['R2'] += rouge_f1(pred, ref, ngram=2)
        scores['RL'] += rouge_f1(pred, ref, ngram='LCS')
        scores['BERTSCORE'] += bertscore_f1(pred, ref, model='bert-base-multilingual-cased', layer=9)
    return {k: v / len(predictions) for k, v in scores.items()}

Common pitfalls

• ROUGE scores heavily penalize abstractive summaries due to n-gram mismatch, which may not reflect factual accuracy or fluency.
• BERTSCORE must be computed using bert-base-multilingual-cased at layer 9 to match the paper's reported values.
• The 'Xtreme' test set contains more abstract summaries, causing a significant performance drop compared to the 'canonical' test set.

Evidence (verbatim from paper)

We use three ROUGE (Lin, 2004) F-1 scores as evaluation metrics: R1 (unigram overlap), R2 (bigram overlap), and RL (longest common subsequence overlap). In addition, we also provide BERTSCORE (F-1), as has recently been used for machine translation evaluation (Zhang et al., 2020b). We use the development set to select the best checkpoint during training, and report the evaluation scores for the canonical and Xtreme test sets in Table 4.

Citation

@misc{koto2020liputan6,
  title={Liputan6: A Large-scale Indonesian Dataset for Text Summarization},
  author={Koto et al. (2020)},
  year={2020},
  note={arXiv:2011.00679}
}

• arXiv: 2011.00679