Bouquet Mt Eval

bouquet-mt-eval

BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation — Andrews et al. (2025) (arXiv:2502.04314, 2025)

What this evaluates

Evaluates machine translation systems on a contamination-free, multilingual dataset covering diverse domains and registers. It measures translation quality at both sentence and paragraph levels to assess how well models handle linguistic diversity and cultural authenticity across 8 major languages.

Datasets

• BOUQuET — total ?; splits: test (-1)

Metrics

• CometKiwi (primary) — range: [0, 1]
  • Neural MT quality metric predicting human-like scores on a 0-1 scale, where higher values indicate better translation quality. Uses the CometKiwi-da-xl model.
• MetricX — range: [0, 25]
  • Hybrid MT quality metric predicting Direct Assessment scores on a 0-25 scale, where lower values indicate better translation quality. Uses the MetricX-24-hybrid-xl-v2p6 model.

Input / output format

Input: Source sentence or paragraph in one of the 9 source languages (8 BOUQuET languages plus English) paired with a reference translation.

Output: Model-generated translation in the target language.

Scoring recipe

def evaluate(system, dataset, level='sentence'):
    com_scores, metx_scores = [], []
    for src, ref, pred in dataset[level]:
        com_scores.append(cometkiwi_da_xl.predict(src, pred))
        metx_scores.append(metricx_24.predict(src, pred))
    avg_com = mean(com_scores)
    avg_metx = mean(metx_scores)
    ranking = rank_systems_by(com_scores)
    return {'COM': avg_com, 'MetX': avg_metx, 'ranking': ranking}
# Compare rankings across datasets using swap count and pearson correlation

Common pitfalls

• Evaluating at sentence-level versus paragraph-level yields significantly different system rankings and lower Pearson correlation, so the evaluation level must be explicitly reported.
• MetricX uses an inverse scale (lower is better) compared to CometKiwi (higher is better), which can cause confusion when comparing or ranking systems across datasets.
• The dataset covers 8 languages plus English as a pivot, so evaluation directions must be specified and averaged correctly across all 9 source languages.

Evidence (verbatim from paper)

Following the official evaluation metrics of WMT 2024 (Kocmi et al., [2024]), we use two automatic metrics: CometKiwi (CometKiwi-da-xl, range 0-1 and ↑ better, COM) (Chimoto and Bassett, [2022]) and MetricX (MetricX-24-hybrid-xl-v2p6, range 0-25 and ↓ better, MetX) (Juraska et al., [2024]). We include in the benchmarking datasets that cover Source-BOUQuET languages (FLORES+ and NTREX-128).

Citation

@misc{andrews2025bouquet,
  title={BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation},
  author={Andrews et al. (2025)},
  year={2025},
  note={arXiv:2502.04314}
}

• arXiv: 2502.04314