Sentencepiece Tokenizer

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

bishal-pdMSFT Updated 0 repo stars

File contents

bishal-pdMSFT/msb-skills commit b59f870590

Frequently asked questions

npx get-skill add bishal-pdmsft/sentencepiece-tokenizer