Tokenizer Task Eval

tokenizer-task-eval

Tokenization is Sensitive to Language Variation — Wegmann et al. (2025) (arXiv:2502.15343, 2025)

What this evaluates

Evaluates how different tokenizer configurations (pre-tokenizer, fitting corpus, vocabulary size) affect downstream BERT performance on tasks requiring robustness vs. sensitivity to language variation.

Datasets

• AV (Authorship Verification) — total ?; splits: test (-1)
• PAN — total ?; splits: test (-1)
• CORE — total ?; splits: test (-1)
• NUCLE — total ?; splits: test (-1)
• Dialect — total ?; splits: test (-1)
• GLUE — total ?; splits: test (-1)
• GLUE+typo — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of total instances.
• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, computed per task (macro or micro average as standard for these benchmarks).

Input / output format

Input: Text instances from each dataset, tokenized using varying pre-tokenizers (e.g., NO, WS, _WS, LLAMA3, GPT2), fitting corpora (PMed, Wiki, Twitter), and vocabulary sizes (500, 4k, 32k, 64k, 128k), then fed into fine-tuned BERT models.

Output: Predicted class labels for each downstream task (e.g., authorship, dialect, grammatical error, NLI).

Scoring recipe

def score(predictions, gold, task):
    if task in ['AV', 'PAN', 'CORE', 'GLUE', 'GLUE+typo']:
        return accuracy_score(gold, predictions)
    elif task in ['NUCLE', 'Dialect']:
        return f1_score(gold, predictions, average='macro')
    # Average across tasks for AVG column
    # Significance tested via McNemar's test with Bonferroni correction

Common pitfalls

• Assuming optimal tokenizer settings (e.g., vocabulary size) are consistent across all tasks; they vary by task type.
• Overlooking the impact of pre-tokenizer design, which the paper finds has the greatest influence on performance.
• Ignoring statistical significance; performance differences are often small and require McNemar's test with Bonferroni correction to validate.

Evidence (verbatim from paper)

| Model | AV (acc) ↑ | PAN (acc) ↑ | CORE (acc) ↑ | NUCLE (F1) ↑ | Dialect (F1) ↑ | AVG |

Citation

@misc{wegmann2025tokenization,
  title={Tokenization is Sensitive to Language Variation},
  author={Wegmann et al. (2025)},
  year={2025},
  note={arXiv:2502.15343}
}

• arXiv: 2502.15343