Tokenizers 0 23 1

Fast state-of-the-art tokenizers library for NLP written in Rust with Python, Node.js, and Ruby bindings. Use when training custom vocabularies, implementing BPE/WordPiece/Unigram tokenization, building NLP pipelines, or working with transformer models requiring efficient text preprocessing with alignment tracking.

tangledgroup Updated 1 repo stars

File contents

tangledgroup/tangled-skills commit 87da5089d4

Frequently asked questions

npx get-skill add tangledgroup/tokenizers-0-23-1