Conditional Unigram Tokenization Eval

conditional-unigram-tokenization-eval

Conditional Unigram Tokenization with Parallel Data — Vico & Libovický (2025) (arXiv:2507.07824, 2025)

What this evaluates

Evaluates a conditional unigram tokenizer's cross-lingual alignment quality and its impact on downstream machine translation and language modeling tasks. It measures intrinsic tokenization properties, alignment accuracy, and task-specific performance metrics.

Datasets

• NLLB — total 51000000; splits: train (51000000)
• MultiParaCrawl — total 483000; splits: train (483000)
• WMT2020 — total 60000; splits: train (60000)
• Flores — total ?; splits: test (-1)
• WMT2020 test set — total ?; splits: test (-1)

Metrics

• Parity — range: ratio
  • Ratio of tokens produced by the target tokenizer to tokens produced by the reference source tokenizer. Lower values indicate better cross-lingual sequence length alignment.
• Fertility — range: ratio
  • Average number of tokens produced per word. Lower values (minimum 1.0) indicate that words remain coherent semantic units.
• One-to-one — range: [0, 1]
  • Proportion of source tokens that have exactly one aligned target token, which is also aligned to exactly one source token. Measured on the source side. Higher is better.
• Unaligned — range: [0, 1]
  • Portion of source tokens that are not aligned to any target tokens. Measured on the source sequence. Lower is better.
• chrF++ (primary) — range: percent
  • Character n-gram F-score used for machine translation evaluation. Higher scores indicate better translation quality.
• Perplexity per byte — range: ratio
  • Language model perplexity normalized by the byte length of the text, allowing fair comparison across different vocabulary sizes. Lower is better.

Input / output format

Input: Parallel source-target sentence pairs for tokenizer training and intrinsic evaluation; monolingual target sentences for language modeling evaluation.

Output: Tokenized sequences for intrinsic/alignment metrics; translated text for MT evaluation; next-token probability distributions for language modeling.

Scoring recipe

def compute_parity(pred_tokens, ref_tokens):
    return len(pred_tokens) / len(ref_tokens)

def compute_chrf(hypotheses, references):
    import sacrebleu
    return sacrebleu.corpus_chrf(hypotheses, references).score

def compute_fertility(pred_tokens, ref_words):
    return len(pred_tokens) / len(ref_words)

def compute_one_to_one(alignments):
    for s, t in alignments if s == t == 1)
    return one_to_one / len(alignments)

Common pitfalls

• Parity and Fertility scores are highly sensitive to vocabulary size and language family similarities, not just tokenizer quality.
• Alignment metrics (One-to-one, Unaligned) rely on an external aligner (Eflomal) and dev-set priors, which can introduce bias if the dev set distribution differs from the test set.
• Perplexity per byte normalizes by byte length, making it incomparable to standard token-level perplexity.

Evidence (verbatim from paper)

We assess tokenization quality using the following metrics: Parity (↓). This measures the ratio of tokens produced by our tokenizer in the target language to those produced by the reference tokenizer in the source language (Petrov et al., [2023]). Optimal tokenization should yield similar sequence lengths across languages. Fertility (↓). This measures the average number of tokens per word (Rust et al., [2021]). Lower fertility (minimum 1.0) indicates that words remain coherent semantic units. We evaluate models using chrF++ on Flores test sets (and WMT2020 test set for German-Upper Sorbian)... we use perplexity per byte to compare models with different vocabularies.

Citation

@misc{vico2025conditional,
  title={Conditional Unigram Tokenization with Parallel Data},
  author={Vico & Libovický (2025)},
  year={2025},
  note={arXiv:2507.07824}
}

• arXiv: 2507.07824