Swiltra Bench Eval

swiltra-bench-eval

SwiLTra-Bench: The Swiss Legal Translation Benchmark — Niklaus et al. (2025) (arXiv:2503.01372, 2025)

What this evaluates

Evaluates large language models and specialized translation systems on their ability to accurately translate Swiss legal documents (laws, headnotes, press releases) across four national languages and English. It probes domain-specific translation quality, contextual understanding, and zero-shot versus fine-tuned performance in a legal context.

Datasets

• SwiLTra-Bench — total 180000; splits: train (-1), val (-1), test (-1); repo https://github.com/JoelNiklaus/SwissLegalTranslations

Metrics

• GEMBA-MQM (primary) — range: [0, 1]
  • Generative Multi-dimensional Quality Metrics, a model-based metric that predicts human MQM (Mutual Quality Measurement) error severity scores. It outputs a continuous score where higher values indicate better translation quality.
• BLEU — range: [0, 1]
  • Bilingual Evaluation Understudy, a lexical metric computing n-gram precision between reference and hypothesis translations, typically with a brevity penalty.
• ChrF — range: [0, 1]
  • Character n-gram F-score, measuring the harmonic mean of character n-gram precision and recall between reference and hypothesis.
• METEOR — range: [0, 1]
  • Metric for Evaluation of Translation with Explicit ORdering, aligning words using exact match, stem, synonym, and paraphrase matches to compute a weighted F-score.
• BERTScore — range: [0, 1]
  • Computes semantic similarity using contextual embeddings from BERT, calculating precision, recall, and F1 based on cosine similarity of token representations.
• XCOMET — range: [0, 1]
  • Cross-lingual Optimized Metric for Evaluation of Translation, a reference-free or reference-based model that predicts translation quality scores based on source, hypothesis, and context embeddings.
• BLEURT — range: [0, 1]
  • BERT-based Evaluation Utility for Ranking Translations, a learned metric fine-tuned on human judgments to output a quality score between 0 and 1.

Input / output format

Input: Source text in a specific language (Swiss German, French, Italian, Romansh, or English) paired with a target language instruction. For fine-tuning, formatted as a table-like prompt: | {source language}: | {source text} | \n | --- | --- | \n | {target language}: | {target text} |. For zero-shot evaluation, standard instruction prompts are used.

Output: Translated text in the target language.

Scoring recipe

def score(predictions, references):
    results = {}
    for pred, ref in zip(predictions, references):
        results['bleu'] += nltk.translate.bleu_score.sentence_bleu([ref], pred)
        results['chrF'] += chrF_score(ref, pred)
        results['meteor'] += meteor_score([ref], pred)
        results['bertscore'] += bertscore_score(ref, pred)
        results['xcomet'] += xcomet_score(ref, pred)
        results['gemba_mqm'] += gemba_mqm_score(ref, pred)
    return {k: v / len(predictions) for k, v in results.items()}

Common pitfalls

• BLEURT and XCOMET have a strict 512-token limit and cannot process press releases, requiring truncation or exclusion from evaluation.
• Fine-tuning requires replicating the exact instruction template and chat format used in the paper to ensure fair comparison with open SLMs.
• GEMBA-MQM is prioritized for human alignment but is computationally intensive and may vary based on the underlying annotator model configuration.

Evidence (verbatim from paper)

We evaluated translations using lexical (BLEU Papineni et al. ([2002]), ChrF Popović ([2015]), METEOR Banerjee and Lavie ([2005])) and model-based metrics (BERTScore Zhang et al. ([2020]), BLEURT Sellam et al. ([2020]), XCOMET Guerreiro et al. ([2024]), GEMBA-MQM Kocmi and Federmann ([2023])). Due to the 512-token limit, BLEURT and XCOMET cannot process press releases. Given GEMBA-MQM’s strong correlation with human judgments, we prioritized it alongside XCOMET, METEOR, and ChrF, ensuring both lexical and trained metrics for diversity.

Citation

@misc{niklaus2025swiltra,
  title={SwiLTra-Bench: The Swiss Legal Translation Benchmark},
  author={Niklaus et al. (2025)},
  year={2025},
  note={arXiv:2503.01372}
}

• arXiv: 2503.01372