QA Translation Fidelity Eval

qa-translation-fidelity-eval

Science Across Languages: Assessing LLM Multilingual Translation of Scientific Papers — Kleidermacher & Zou (2025) (arXiv:2502.17882, 2025)

What this evaluates

This benchmark evaluates how well LLM-generated translations preserve the scientific content of original papers. It measures translation fidelity by testing whether a reading model can accurately answer comprehension questions derived from the source text, using only the translated version as context.

Datasets

• Science Across Languages QA Benchmark — total 6; splits: test (6)

Metrics

• quiz accuracy (primary) — range: percent
  • Calculated as the percentage of correctly answered multiple-choice questions out of the total 50 questions per article. Formula: (correct_answers / 50) * 100.

Input / output format

Input: For each instance (article + target language): the full translated article text (preserving JATS XML or plain text) and a set of 50 multiple-choice questions translated into the target language.

Output: A list of 50 answers, each selecting one option from A, B, C, D, E, or 'I don't know'.

Scoring recipe

def compute_quiz_accuracy(predictions, gold_answers):
    # predictions: list of 50 model answers (str)
    # gold_answers: list of 50 correct options (str)
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
    return (correct / len(gold_answers)) * 100

Common pitfalls

• The evaluation model may rely on pre-training knowledge rather than the translated text; the protocol mitigates this by filtering out articles where the model scores 0% without context and by omitting the quiz-generation prompt during evaluation.
• Quiz questions or answer choices can be ambiguous, causing minor accuracy drops even for high-quality translations; the authors note this affects the English baseline as well.
• Translating the quiz questions into the target language is required for valid results; using untranslated English questions yields different scores (see Appendix A.1.1).

Evidence (verbatim from paper)

The model’s quiz accuracy, graded against the answer key, constitutes the benchmark result. For instance, if the LLM correctly answers 48 out of 50 questions for a given article in a particular language, the benchmark score for that translation would be 96%.

Citation

@misc{kleidermacher2025scienceacross,
  title={Science Across Languages: Assessing LLM Multilingual Translation of Scientific Papers},
  author={Kleidermacher & Zou (2025)},
  year={2025},
  note={arXiv:2502.17882}
}

• arXiv: 2502.17882