qhjqhj00 subword-fertility-pcw
Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi — Fatimah et al. (2026) (arXiv:2603.03508, 2026)
What this evaluates
Evaluates the efficiency and compactness of subword tokenizers on Hindi, English, and code-mixed text by measuring how many tokens are generated per word and how frequently words are split into multiple tokens.
Datasets
- Tokenizer Test Set — total 31500; splits: test (31500)
Metrics
Subword Fertility (SF)(primary) — range: other (tokens/word)- Average number of tokens generated per word. Calculated as total tokens divided by total words. Lower values indicate more compact representations.
Proportion of Continued Words (PCW)— range: [0, 1]- Fraction of words that are split into two or more tokens. Calculated as the count of split words divided by total words. Lower values suggest less aggressive splitting.
Input / output format
Input: A list of 31,500 words (15,000 Hindi, 15,000 English, 1,500 code-mixed) from the evaluation set.
Output: Tokenized sequence for each input word.
Scoring recipe
def compute_metrics(words, tokens_list):
total_words = len(words)
total_tokens = sum(len(toks) for toks in tokens_list)
split_words = sum(1 for toks in tokens_list if len(toks) > 1)
fertility = total_tokens / total_words
pcw = split_words / total_words
return {'SF': fertility, 'PCW': pcw}
Common pitfalls
- Subword Fertility and PCW can trade off; a tokenizer might have low PCW but high fertility (e.g., HindRoBERTa), so optimizing for one metric alone may not yield the most efficient tokenizer.
- The test set composition is unbalanced (15k Hindi, 15k English, 1.5k code-mixed), which may skew overall metrics toward English performance and underrepresent code-mixed behavior.
- UNK rate is reported in the comparison table but not defined in the text, making it unclear how unknown tokens are counted or handled during evaluation.
Evidence (verbatim from paper)
To assess tokenizer performance, we relied on two standard metrics:
•
Subword Fertility (SF): the average number of tokens per word. Lower values indicate more compact representations.
•
Proportion of Continued Words (PCW): the fraction of words that are split into two or more tokens. Lower values suggest less aggressive splitting.
We evaluated the trained tokenizer on a test set of 31,500 words, including 15,000 Hindi words, 15,000 English words, and 1,500 code-mixed words.
Citation
@misc{fatimah2026lilmoo,
title={Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi},
author={Fatimah et al. (2026)},
year={2026},
note={arXiv:2603.03508}
}
- arXiv: 2603.03508