# Flores200 Mt Eval

> Evaluates multilingual machine translation quality across 60 languages and 234 translation directions. It specifically probes a model's ability to handle high-, medium-, and low-resource languages while mitigating directional degeneration in symmetric multi-way translation. Use when the user wants to benchmark on FLORES-200, or asks about evaluating this task. Reports COMET-22.

- Skill: `qhjqhj00/flores200-mt-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/flores200-mt-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/flores200-mt-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/flores200-mt-eval

---


# flores200-mt-eval

> Beyond English: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs — Luo et al. (2025) (arXiv:2511.07003, 2025)

## What this evaluates

Evaluates multilingual machine translation quality across 60 languages and 234 translation directions. It specifically probes a model's ability to handle high-, medium-, and low-resource languages while mitigating directional degeneration in symmetric multi-way translation.

## Datasets

- **FLORES-200** — total 2500; splits: devtest (2500); HF `flores`

## Metrics

- `COMET-22` **(primary)** — range: percent (0-100)
  - A neural reference-based metric that uses a multilingual encoder (XLM-R) to compute contextual representations of source, reference, and hypothesis, followed by a cross-attention mechanism and a regression head to predict a quality score. Scores are scaled to a 0-100 range for reporting.

## Input / output format

**Input**: Source sentence in language X, optionally prefixed with explicit direction tags and a target-language separator.

**Output**: Target sentence in language Y.

## Scoring recipe

```python
import comet
model = comet.load_model('Unbabel/comet-22')
scores = []
for src, ref, hyp in zip(sources, references, hypotheses):
    score = model.predict({'src': src, 'ref': ref, 'hyp': hyp})
    scores.append(score * 100)
return sum(scores) / len(scores)
```

## Common pitfalls

- Evaluating on language pairs not supported by both the baseline and the proposed model, which skews cross-model comparisons.
- Overlooking directional asymmetry, as performance often differs significantly between forward (En/Zh→X) and reverse (X→En/Zh) translation directions.
- Ignoring the custom Chinese-Mongolian testset, which was created via native annotators to fill a gap in FLORES-200.

## Evidence (verbatim from paper)

> We evaluate on FLORES-200 devtest. We adopt COMET-22 as our primary evaluation metric, and report SacreBLEU in the Appendix. For fairness, evaluation is conducted on the intersection of language pairs supported by each baseline and LMT, covering Chinese-centric directions when applicable.

## Citation

```bibtex
@misc{luo2025beyondenglish,
  title={Beyond English: Toward Inclusive and Scalable Multilingual Machine Translation with LLMs},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2511.07003}
}
```

- arXiv: 2511.07003