# Kreyol Mt Eval

> Evaluates machine translation performance across 41 Creole languages, testing cross-lingual transfer and the impact of data cleaning and scale on translation quality. Use when the user wants to benchmark on Kreyol-MT, or asks about evaluating this task. Reports BLEU.

- Skill: `qhjqhj00/kreyol-mt-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/kreyol-mt-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/kreyol-mt-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/kreyol-mt-eval

---


# kreyol-mt-eval

> Krey\`ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages — Robinson et al. (2024) (arXiv:2405.05376, 2024)

## What this evaluates

Evaluates machine translation performance across 41 Creole languages, testing cross-lingual transfer and the impact of data cleaning and scale on translation quality.

## Datasets

- **Kreyol-MT** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/JHU-CLSP/Kreyol-MT

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard sentence-level BLEU score (Papineni et al., 2002), computed with tokenizer normalization. (Note: exact metric not explicitly stated in the provided section, but BLEU is the standard evaluation metric for machine translation.)

## Input / output format

**Input**: Source language Creole sentence.

**Output**: Target language Creole sentence.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    # Tokenize predictions and references using standard MT tokenizer
    # Compute n-gram precision for n=1..4
    # Apply brevity penalty based on reference and prediction lengths
    # Return geometric mean of precisions * brevity penalty
    # Note: Exact implementation details are not provided in the text.
```

## Common pitfalls

- Test set contamination: must explicitly remove any sentence pairs from train/dev that overlap with pre-existing test sets.
- Cleaning configuration mismatch: models trained on cleaned vs. non-cleaned data use different test splits, making direct comparison invalid.
- Zero-shot evaluations: language pairs with removed training data are still evaluated zero-shot, which may skew aggregate results.

## Evidence (verbatim from paper)

> After filtering, we prepare a train/dev/test split for each language pair of the remaining data by aggregating all sentences and splitting randomly with a fixed random seed, and a target ratio of 85 / 5 / 10, with minimum 50 and maximum 2000 sentences for the dev and test sets. We discard train sets with fewer than 100 sentences.

## Citation

```bibtex
@misc{robinson2024kreyolmt,
  title={Krey\`ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages},
  author={Robinson et al. (2024)},
  year={2024},
  note={arXiv:2405.05376}
}
```

- arXiv: 2405.05376