Tokenizer Fix Python

Fix Python conversion issues in OpenVINO Tokenizers. Use when: tokenizer-diagnostics reports root_cause_location=python, unsupported types need new handlers in hf_parser.py, pipeline step mapping or merging is incorrect, tokenizer_pipeline.py step classes need fixes.

openvinotoolkit Updated 11.2k repo stars

File contents

openvinotoolkit/openvino_tokenizers commit 70bd27c4c0

Frequently asked questions

npx get-skill add openvinotoolkit/tokenizer-fix-python