qhjqhj00 kreyol-mt-eval
Krey`ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages — Robinson et al. (2024) (arXiv:2405.05376, 2024)
What this evaluates
Evaluates machine translation performance across 41 Creole languages, testing cross-lingual transfer and the impact of data cleaning and scale on translation quality.
Datasets
- Kreyol-MT — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/JHU-CLSP/Kreyol-MT
Metrics
BLEU(primary) — range: [0, 100]- Standard sentence-level BLEU score (Papineni et al., 2002), computed with tokenizer normalization. (Note: exact metric not explicitly stated in the provided section, but BLEU is the standard evaluation metric for machine translation.)
Input / output format
Input: Source language Creole sentence.
Output: Target language Creole sentence.
Scoring recipe
def compute_bleu(predictions, references):
# Tokenize predictions and references using standard MT tokenizer
# Compute n-gram precision for n=1..4
# Apply brevity penalty based on reference and prediction lengths
# Return geometric mean of precisions * brevity penalty
# Note: Exact implementation details are not provided in the text.
Common pitfalls
- Test set contamination: must explicitly remove any sentence pairs from train/dev that overlap with pre-existing test sets.
- Cleaning configuration mismatch: models trained on cleaned vs. non-cleaned data use different test splits, making direct comparison invalid.
- Zero-shot evaluations: language pairs with removed training data are still evaluated zero-shot, which may skew aggregate results.
Evidence (verbatim from paper)
After filtering, we prepare a train/dev/test split for each language pair of the remaining data by aggregating all sentences and splitting randomly with a fixed random seed, and a target ratio of 85 / 5 / 10, with minimum 50 and maximum 2000 sentences for the dev and test sets. We discard train sets with fewer than 100 sentences.
Citation
@misc{robinson2024kreyolmt,
title={Krey\`ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages},
author={Robinson et al. (2024)},
year={2024},
note={arXiv:2405.05376}
}
- arXiv: 2405.05376