Tokenizer Diagnostics

Diagnose tokenizer conversion issues for OpenVINO Tokenizers. Use when: tokenizer-checker reports failures, need to pinpoint root cause location (Python conversion vs C++ operation), identify which pipeline stage diverges, determine whether to use tokenizer-fix-python or tokenizer-fix-cpp skill.

openvinotoolkit Updated 11.2k repo stars

File contents

openvinotoolkit/openvino_tokenizers commit cb47b0ba85

Frequently asked questions

npx get-skill add openvinotoolkit/tokenizer-diagnostics