Smol Chrf Eval

smol-chrf-eval

SMOL: Professionally translated parallel data for 115 under-represented languages — Caswell et al. (2025) (arXiv:2502.12301, 2025)

What this evaluates

Evaluates machine translation quality for 115 under-represented languages using professionally translated parallel data. It measures the improvement in character-level n-gram F-score (ChrF) after fine-tuning a baseline model on the Smol dataset compared to the unfine-tuned baseline.

Datasets

• SmolSent — total ?; splits: test (-1)
• SmolDoc — total ?; splits: test (-1)

Metrics

• ChrF (primary) — range: [0, 100]
  • Character n-gram F-score, computed as the harmonic mean of character-level precision and recall across n-grams (typically up to 6-grams).

Input / output format

Input: Source sentence or document in English.

Output: Translated sentence or document in the target language.

Scoring recipe

def compute_chrf(hypotheses, references, n_max=6):
    hyp_ngrams = [get_char_ngrams(h, n_max) for h in hypotheses]
    ref_ngrams = [get_char_ngrams(r, n_max) for r in references]
    matches = sum(min(h.count(g), r.count(g)) for h, r in zip(hyp_ngrams, ref_ngrams))
    prec = matches / sum(len(h) for h in hyp_ngrams)
    rec = matches / sum(len(r) for r in ref_ngrams)
    return 2 * (prec * rec) / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• The paper reports $\Delta_{FT}$ (baseline ChrF minus fine-tuned ChrF) as the primary sorting metric, not absolute ChrF.
• ChrF operates at the character level, making it sensitive to punctuation, casing, and tokenization differences.
• Results are presented per language pair; macro-averaging across languages may obscure performance on specific language families or scripts.

Evidence (verbatim from paper)

Results are sorted by the $\Delta_{FT}$, which is the ChrF of the Both model minus the ChrF of the finetuned Both model—in other words, how much the finetuning on Smol improved the baseline model.

Citation

@misc{caswell2025smol,
  title={SMOL: Professionally translated parallel data for 115 under-represented languages},
  author={Caswell et al. (2025)},
  year={2025},
  note={arXiv:2502.12301}
}

• arXiv: 2502.12301