Crosssum Alignment Eval

crosssum-alignment-eval

CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairs — Abhik Bhattacharjee et al. (2021) (arXiv:2112.08804, 2021)

What this evaluates

Evaluates the quality of automatically induced cross-lingual summary alignments in the CrossSum dataset by measuring human agreement on whether two summaries correspond to the same source article.

Datasets

• CrossSum — total ?; splits: alignment_eval (1800); repo https://github.com/csebuetnlp/CrossSum

Metrics

• alignment_accuracy (primary) — range: [0, 1]
  • Proportion of summary pairs where both annotators answer 'yes' to whether they summarize the same article.
• cohen_kappa — range: [-1, 1]
  • Cohen's Kappa statistic measuring inter-annotator agreement on the binary alignment labels.

Input / output format

Input: Two summary sequences in different languages (x, y), optionally accompanied by their source articles.

Output: Binary yes/no judgment from each of two bilingual annotators per pair.

Scoring recipe

def compute_metrics(annotations):
    # annotations: list of (annotator1_label, annotator2_label)
    correct = sum(1 for a1, a2 in annotations if a1 == 'yes' and a2 == 'yes')
    alignment_accuracy = correct / len(annotations)
    labels1 = [a1 for a1, a2 in annotations]
    labels2 = [a2 for a1, a2 in annotations]
    cohen_kappa = compute_cohen_kappa(labels1, labels2)
    return alignment_accuracy, cohen_kappa

Common pitfalls

• Non-English pairs are evaluated via a pivoting approach through English, which assumes transitive alignment quality and may not reflect direct cross-lingual fidelity.
• The evaluation only checks summary-to-summary correspondence, explicitly excluding article-summary correctness to reduce costs.
• Requires expert bilingual annotators for each target language paired with English, making it infeasible for distant language pairs without the English pivot.

Evidence (verbatim from paper)

We hired bilingually proficient expert annotators adept in the language of interest and English. Two annotators labeled each language pair where one language is English. We presented them with corresponding summaries of the cross-lingual pairs (and optionally the articles themselves) and elicited yes/no answers to the question: “Can the provided sequences be considered summaries for the same article?” ... We deem a sequence pair accurate if both annotators judge it as valid. ... We used Cohen’s Kappa (Cohen, 1960) to establish the inter-annotator agreement and show the corresponding statistics in Table 3 in the Appendix.

Citation

@misc{bhattacharjee2021crosssum,
  title={CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairs},
  author={Abhik Bhattacharjee et al. (2021)},
  year={2021},
  note={arXiv:2112.08804}
}

• arXiv: 2112.08804