# Tokenizer Task Eval

> Evaluates how different tokenizer configurations (pre-tokenizer, fitting corpus, vocabulary size) affect downstream BERT performance on tasks requiring robustness vs. sensitivity to language variation. Use when the user wants to benchmark on AV (Authorship Verification), PAN, CORE, NUCLE, Dialect, GLUE, GLUE+typo, or asks about evaluating this task. Reports accuracy, F1.

- Skill: `qhjqhj00/tokenizer-task-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tokenizer-task-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tokenizer-task-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tokenizer-task-eval

---


# tokenizer-task-eval

> Tokenization is Sensitive to Language Variation — Wegmann et al. (2025) (arXiv:2502.15343, 2025)

## What this evaluates

Evaluates how different tokenizer configurations (pre-tokenizer, fitting corpus, vocabulary size) affect downstream BERT performance on tasks requiring robustness vs. sensitivity to language variation.

## Datasets

- **AV (Authorship Verification)** — total ?; splits: test (-1)
- **PAN** — total ?; splits: test (-1)
- **CORE** — total ?; splits: test (-1)
- **NUCLE** — total ?; splits: test (-1)
- **Dialect** — total ?; splits: test (-1)
- **GLUE** — total ?; splits: test (-1)
- **GLUE+typo** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances out of total instances.
- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, computed per task (macro or micro average as standard for these benchmarks).

## Input / output format

**Input**: Text instances from each dataset, tokenized using varying pre-tokenizers (e.g., NO, WS, _WS, LLAMA3, GPT2), fitting corpora (PMed, Wiki, Twitter), and vocabulary sizes (500, 4k, 32k, 64k, 128k), then fed into fine-tuned BERT models.

**Output**: Predicted class labels for each downstream task (e.g., authorship, dialect, grammatical error, NLI).

## Scoring recipe

```python
def score(predictions, gold, task):
    if task in ['AV', 'PAN', 'CORE', 'GLUE', 'GLUE+typo']:
        return accuracy_score(gold, predictions)
    elif task in ['NUCLE', 'Dialect']:
        return f1_score(gold, predictions, average='macro')
    # Average across tasks for AVG column
    # Significance tested via McNemar's test with Bonferroni correction
```

## Common pitfalls

- Assuming optimal tokenizer settings (e.g., vocabulary size) are consistent across all tasks; they vary by task type.
- Overlooking the impact of pre-tokenizer design, which the paper finds has the greatest influence on performance.
- Ignoring statistical significance; performance differences are often small and require McNemar's test with Bonferroni correction to validate.

## Evidence (verbatim from paper)

> | Model | AV (acc) ↑ | PAN (acc) ↑ | CORE (acc) ↑ | NUCLE (F1) ↑ | Dialect (F1) ↑ | AVG |

## Citation

```bibtex
@misc{wegmann2025tokenization,
  title={Tokenization is Sensitive to Language Variation},
  author={Wegmann et al. (2025)},
  year={2025},
  note={arXiv:2502.15343}
}
```

- arXiv: 2502.15343