Liveclktb Eval

liveclktb-eval

LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs — Guo et al. (2025) (arXiv:2511.14774, 2025)

What this evaluates

Evaluates multilingual LLMs on their ability to transfer factual knowledge across languages using time-sensitive, real-world events that occur after the model's training cutoff. It measures both in-language factual recall and cross-lingual generalization performance across domains like music, movies, and sports.

Datasets

• LiveCLKTBench — total ?; splits: test (-1)

Metrics

• Overall Score — range: [0, 1]
  • Accuracy of model-generated answers against gold answers when evaluated in the source language.
• Transfer Score (primary) — range: [0, 1]
  • Accuracy of model-generated answers against gold answers when evaluated across different source-target language pairs to measure cross-lingual generalization.

Input / output format

Input: Factual question-answer pairs grounded in source documents, presented in a target language for cross-lingual evaluation.

Output: Model-generated text answers to the questions.

Scoring recipe

def compute_scores(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    overall_score = correct / len(golds)
    # Transfer score aggregates accuracy specifically for cross-lingual instances
    transfer_score = correct / len(golds)  # Computed on cross-lingual subset
    return overall_score, transfer_score

Common pitfalls

• Assuming cross-lingual transfer performance is symmetric across language pairs, whereas the benchmark shows strong directional asymmetry (e.g., en->ja vs ja->en).
• Overestimating the benefits of model scaling, as gains show diminishing returns and saturate at higher parameter counts.
• Ignoring domain-specific variance, as sports consistently yields the weakest performance compared to music and movies.

Evidence (verbatim from paper)

Table[2] presents the Overall and Transfer scores for each model across the three domains—music, movies, and sports—as well as their domain averages. Across models, the average Overall Score ranges from 0.221 (OLMo-2-7B) to 0.414 (Gemma-2-9B), while the Transfer Score ranges from 0.567 (OLMo-2-7B) to 0.747 (Qwen2.5-7B).

Citation

@misc{guo2025liveclktb,
  title={LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2511.14774}
}

• arXiv: 2511.14774