Toksuite Eval

toksuite-eval

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior — Altintas et al. (2025) (arXiv:2512.20757, 2025)

What this evaluates

This benchmark evaluates the robustness of language model tokenizers against real-world input perturbations, including orthographic errors, script variations, homoglyphs, diacritics, and stylistic changes across five languages. It isolates the impact of tokenizer design by testing identical model architectures with different tokenization strategies.

Datasets

• TokSuite — total ?; splits: test (-1); repo https://github.com/r-three/Tokenizers

Metrics

• relative performance drop (primary) — range: other
  • Calculated as (Acc_can - Acc_pert) / Acc_can, where Acc_can is accuracy on canonical input and Acc_pert is accuracy on perturbed input. Lower values indicate greater robustness; negative values indicate performance improvement on perturbed inputs.

Input / output format

Input: Perturbed or canonical multiple-choice questions (text + options) in English, Turkish, Italian, Farsi, or Chinese.

Output: Model's predicted answer choice (e.g., A, B, C, D) or generated text response.

Scoring recipe

def compute_relative_drop(acc_can, acc_pert):
    if acc_can == 0:
        return 0.0
    return (acc_can - acc_pert) / acc_can

# For each perturbation category:
acc_can = mean(correct_predictions on canonical inputs)
acc_pert = mean(correct_predictions on perturbed inputs)
metric = compute_relative_drop(acc_can, acc_pert)

Common pitfalls

• Negative metric values indicate performance improvement on perturbed inputs, not a calculation error.
• The metric measures relative drop, so a value of 0.0 means identical performance, while 1.0 means complete failure on perturbed inputs.
• Zero-width characters and homoglyphs affect morphologically complex languages (e.g., Farsi) differently due to whitespace and ligature dependencies.

Evidence (verbatim from paper)

Values represent relative performance drop $(rac{ ext{Acc}{ ext{can}} - ext{Acc}{ ext{pert}}}{ ext{Acc}_{ ext{can}}})$ ; lower values indicate greater robustness.

Citation

@misc{altintas2025toksuite,
  title={TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior},
  author={Altintas et al. (2025)},
  year={2025},
  note={arXiv:2512.20757}
}

• arXiv: 2512.20757