Summarization Metrics Eval

summarization-metrics-eval

Summarization Metrics for Spanish and Basque: Do Automatic Scores and LLM-Judges Correlate with Humans? — Barnes et al. (2025) (arXiv:2503.17039, 2025)

What this evaluates

Evaluates the correlation between automatic summarization metrics and LLM-as-a-Judge models against human judgments across five quality criteria (coherence, consistency, fluency, relevance, 5W1H) in Spanish and Basque.

Datasets

• BASSE — total 2040; splits: test (-1); repo https://github.com/hitz-zentroa/summarization

Metrics

• Spearman's $ ho$ (primary) — range: [-1, 1]
  • System-level rank correlation coefficient between human average scores and automatic/LLM scores. Calculated following established NLG evaluation protocols (Louis and Nenkova, 2013; Fabbri et al., 2021).
• ROUGE — range: [0, 1]
  • Token subsequence overlap between candidate and reference summaries. Includes ROUGE-1 to ROUGE-4, ROUGE-L, and ROUGE-su* variants.
• BertScore — range: [0, 1]
  • Average token-level cosine similarity between greedily aligned tokens using multilingual BERT base uncased. Reports precision, recall, and F1.
• BLEU — range: [0, 1]
  • Corpus-level average n-gram precision with a brevity penalty, using SacreBLEU implementation.
• LLM-Judge Score — range: unknown
  • Numeric score assigned by an LLM judge (e.g., GPT-4o, Prometheus 2, Qwen2.5) for a single criterion, prompted with instructions, rubric, reference, and summary in English.

Input / output format

Input: For automatic metrics: candidate summary and reference summary. For LLM judges: original instruction, annotation rubric, human reference answer, and candidate summary.

Output: Numeric score per summary/criterion for each metric or judge model.

Scoring recipe

def compute_spearman_rho(human_scores, metric_scores):
    n = len(human_scores)
    if n < 2:
        return 0.0
    h_ranks = assign_ranks(human_scores)
    m_ranks = assign_ranks(metric_scores)
    d_sq_sum = sum((h - m) ** 2 for h, m in zip(h_ranks, m_ranks))
    rho = 1 - (6 * d_sq_sum) / (n * (n ** 2 - 1))
    return rho

Common pitfalls

• Metrics require language-specific preprocessing: ROUGE-we needs FastText embeddings, BertScore uses mBERT, CIDEr uses Snowball stemmer, and METEOR requires lowercasing.
• LLM judges must be prompted in English despite evaluating non-English summaries, as models struggle to follow non-English instructions.
• Correlation is computed at the system-level (averaging scores per system), not instance-level, which changes the statistical interpretation and requires careful aggregation.

Evidence (verbatim from paper)

We calculate both Spearman’s ($ ho$) and Kendall’s ($ au$) rank correlation coefficients at system-level following Louis and Nenkova (2013) and Fabbri et al. (2021), but report only Spearman in the subsequent section, as the results do not vary depending on the coefficient.

Citation

@misc{barnes2025summarizationmetrics,
  title={Summarization Metrics for Spanish and Basque: Do Automatic Scores and LLM-Judges Correlate with Humans?},
  author={Barnes et al. (2025)},
  year={2025},
  note={arXiv:2503.17039}
}

• arXiv: 2503.17039