Tokenizer Checker

Validate a HuggingFace tokenizer with OpenVINO Tokenizers and OpenVINO GenAI. Use when: checking if a tokenizer converts and works correctly, verifying tokenizer/detokenizer accuracy, testing normalization steps, checking GenAI Tokenizer compatibility.

openvinotoolkit Updated 11.2k repo stars

File contents

openvinotoolkit/openvino_tokenizers commit 139c2836f4

Frequently asked questions

npx get-skill add openvinotoolkit/tokenizer-checker