Tokenizer Fix Python

OpenVINO Tokenizer Python Fixer

Fixes issues in the Python conversion layer that maps HuggingFace tokenizer pipelines to OpenVINO Tokenizers.

When to Use

• tokenizer-diagnostics reported root_cause_location: python
• Common issue categories:
  • Unsupported type: A HuggingFace step type is missing from the parser maps
  • Incorrect mapping: A supported type produces wrong OV steps (wrong params, missing steps)
  • Merge bug: merge_regex_split_steps incorrectly combines incompatible RegexSplitStep instances during finalize()
  • Missing pipeline step class: A new OV operation needs a Python step wrapper

Inputs

Required:

• model_id: HuggingFace model identifier
• diagnosis: Output from the tokenizer-diagnostics skill, particularly:
  • unsupported_types — which HF types need handlers
  • affected_stages — which pipeline stages diverge
  • description — human-readable root cause summary

Helpful context (from tokenizer-diagnostics output):

• The HF→OV pipeline mapping (Step 2 output)
• Finalized pipeline showing step merges
• Pre-tokenization expected vs actual token splits

Prerequisites

Activate the Python virtual environment before running any commands.

1. Locate the virtual environment — check for common directories at the repository root: .venv/, venv/, env/. Use list_dir to find it. If none is found, ask the user for its location.
2. Activate based on the current platform:
   • Linux/macOS: source <venv_path>/bin/activate
   • Windows (cmd): <venv_path>\Scripts\activate.bat
   • Windows (PowerShell): <venv_path>\Scripts\Activate.ps1

Architecture Overview

Conversion Pipeline

HuggingFace tokenizer
    │
    ▼
TransformersTokenizerPipelineParser (hf_parser.py)
    │  reads tokenizer.json
    │  maps each section via type→handler dicts
    ▼
TokenizerPipeline (tokenizer_pipeline.py)
    │  list of *Step dataclasses
    │  finalize() → merges steps, transforms vocab
    ▼
OV Model (via get_tokenizer_ov_subgraph)
    │  each Step.get_ov_subgraph() creates C++ ops
    ▼
Compiled OpenVINO model

Key Files

Parser Maps (hf_parser.py)

The parser uses four type→handler dictionaries. Each handler takes a step_dict (from tokenizer.json) and returns one or more *Step instances:

The tokenization model (BPE/WordPiece/Unigram/WordLevel) is handled by the tokenization_model() method (~L265), not via a map.

Step Class Hierarchy (tokenizer_pipeline.py)

BasePipelineStep
├── NormalizationStep  (normalizer ops)
│   ├── RegexNormalizationStep
│   ├── NormalizeUnicode
│   ├── CaseFoldStep
│   └── CharsmapStep
├── PreTokenizatinStep  (splitting ops)
│   ├── RegexSplitStep
│   ├── WhitespaceSplitStep
│   └── BytesToCharsStep
├── TokenizationModelStep  (vocabulary lookup)
│   ├── BPETokenizationStep
│   ├── WordPieceTokenizationStep
│   ├── UnigramModelStep
│   └── VocabEncoderStep
├── PostTokenizationStep  (combine/truncate/pad)
│   ├── CombineSegmentsStep
│   ├── TruncationStep
│   └── PaddingStep
└── DecodingStep  (detokenizer ops)
    ├── VocabDecoderStep
    ├── CharsToBytesStep
    ├── FuseStep
    └── ByteFallbackStep

Available C++ Operations (src/ov_extension.cpp)

RegexNormalization, RegexSplit, BPETokenizer, WordpieceTokenizer, BytesToChars, CharsToBytes, CombineSegments, VocabEncoder, VocabDecoder, TrieTokenizer, Truncate, FuzeRagged, ByteFallback, SpecialTokensSplit, CharsMapNormalization, CaseFold, NormalizeUnicode, UnigramTokenizer, UTF8Validate, SentencepieceTokenizer, SentencepieceDetokenizer, SentencepieceStreamDetokenizer, RaggedToDense, RaggedToSparse, RaggedToRagged, StringToHashBucket, StringTensorPack, StringTensorUnpack, RaggedTensorPack, EqualStr

Procedure

Step 1: Extract the tokenizer.json structure

Download and inspect the HF tokenizer's pipeline definition:

from tokenizers import Tokenizer
import json

tok = Tokenizer.from_pretrained("<model_id>")
tj = json.loads(tok.to_str())

# Inspect the failing section
print(json.dumps(tj["pre_tokenizer"], indent=2))  # or normalizer, decoder, etc.

Alternatively, use openvino_tokenizers diagnose <model_id> output which already maps all sections.

Step 2: Identify the fix category

Based on the diagnosis, determine which category the fix falls into:

Category A: Missing type in parser map

Symptom: unsupported_types: [SomeType] in diagnosis output.

Fix pattern:

1. Read the HF tokenizers source or tokenizer.json spec to understand what SomeType does
2. Find or create the corresponding OV step class in tokenizer_pipeline.py
3. Add the handler to the appropriate map in hf_parser.py:

# In hf_parser.py — add to the appropriate map:
normalizers_map["NewType"] = lambda step_dict: SomeStep(
    param1=step_dict.get("param1", "default"),
    param2=step_dict["param2"],
)

If a new step class is needed:

# In tokenizer_pipeline.py — add a new step:
@dataclass
class NewNormalizationStep(NormalizationStep):
    param1: str = ""

    def get_ov_subgraph(self, input_nodes: list[Output]) -> list[Output]:
        # Use an existing C++ op
        input_nodes.extend(create_string_constant_node(self.param1))
        return (
            _get_factory()
            .create("RegexNormalization", input_nodes, {"global_replace": True})
            .outputs()
        )

The get_ov_subgraph pattern:

1. Extend input_nodes with constant parameters (strings via create_string_constant_node, scalars via make_constant_node)
2. Call _get_factory().create("<OpName>", input_nodes, {attributes_dict}) to create the OV C++ operation
3. Return .outputs() — the list of output nodes

Category B: Incorrect parameter parsing

Symptom: Conversion succeeds but outputs differ. Diagnosis shows no unsupported types but pre-tokenization or full pipeline tests fail.

Fix pattern:

1. Read the HF tokenizer.json for the failing step and compare with the parser handler
2. Common issues:
   • Reading wrong key from step_dict (e.g. "String" vs "Regex" in pattern dicts)
   • Missing boolean flags that affect behavior
   • Default values that don't match HF defaults

Example — how handler reads a pattern dict:

# HF tokenizer.json may have:  {"pattern": {"String": "abc"}}  or  {"pattern": {"Regex": "a.*c"}}
pattern = step_dict["pattern"].get("String") or step_dict["pattern"]["Regex"]

Category C: Pipeline merge bug (merge_regex_split_steps)

Symptom: Diagnosis shows ⚠ Pre-tokenization merge: N steps → M steps and pre-tokenization test fails. The merged regex pattern in the finalized pipeline produces different splits than running the original patterns sequentially.

Where: TokenizerPipeline.merge_regex_split_steps() in tokenizer_pipeline.py (~L1498).

How it works: The method iterates over RegexSplitStep instances and tries to combine them via RegexSplitStep.__add__, which joins patterns with | (OR). It merges when invert, behaviour, and max_splits all match.

Common fix approaches:

1. Prevent merge for incompatible patterns: Add a check in RegexSplitStep.__add__ that raises ValueError when patterns can't be safely merged (the merge loop catches ValueError and keeps steps separate):

def __add__(self, other: "RegexSplitStep") -> "RegexSplitStep":
    # ... existing checks ...
    # Example: prevent merging patterns with conflicting quantifiers
    if _patterns_conflict(self.split_pattern, other.split_pattern):
        raise ValueError("Patterns cannot be safely merged")
    return self.__class__(
        split_pattern="|".join((self.split_pattern, other.split_pattern)),
        ...
    )

2. Skip merge entirely for specific step configurations: Add a mergeable flag or check specific pattern signatures that are known to conflict.

3. Fix the merge logic: If the |-join is semantically wrong for certain pattern combinations, the patterns may need wrapping in non-capturing groups (?:pattern1)|(?:pattern2) or the merge should be skipped.

Category D: Step finalization issue

Symptom: Individual step mapping looks correct but the finalized pipeline differs unexpectedly.

Where: TokenizerPipeline.finalize() in tokenizer_pipeline.py (~L1547) and individual Step.finalize() methods.

Key finalization behaviors to be aware of:

• BPETokenizationStep.finalize() (~L665): Removes BytesToCharsStep and CharsToBytesStep from the pipeline when is_byte_level=True, absorbing byte-level encoding into the BPE vocab
• merge_regex_split_steps() (~L1498): Merges compatible RegexSplitStep instances
• del_duplicated_split_steps(): Removes duplicate whitespace splitters
• update_metaspace_step_with_special_tokens(): Patches metaspace regex when special tokens exist

Step 3: Implement the fix

1. Read the relevant source files to understand the current implementation
2. Apply the minimal fix — do not refactor surrounding code
3. Ensure the fix handles edge cases present in the test strings:
   • Whitespace variants: \t, \n, multiple spaces
   • Empty strings
   • Unicode: emoji, CJK, accented characters, RTL
   • Mixed content: digits + letters + punctuation

Step 4: Verify

After applying the fix, rebuild and test:

# Rebuild (triggers CMake for C++ extensions)
pip install --pre -Ue . --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

# Run targeted check
openvino_tokenizers check <model_id> [flags]

# Run diagnose to verify pre-tokenization now matches
openvino_tokenizers diagnose <model_id> [flags]

# Run unit tests for the affected operation
python -m pytest tests/layer_tests.py -k <relevant_test> -v

# Run integration tests for the model type
python -m pytest tests/tokenizers_test.py -k <relevant_test> -v

# Run the full test suite to check for regressions
python -m pytest tests/ -v

Step 5: Check for regressions

The fix must not break other tokenizers. Key model sets to verify:

• WordPiece models: bert-base-multilingual-cased, google/mobilebert-uncased
• BPE models: openai-community/gpt2, roberta-base, NousResearch/Meta-Llama-3-8B-Instruct
• SentencePiece models: NousResearch/Llama-2-13b-hf, microsoft/Phi-3-mini-128k-instruct
• TikToken models: Qwen/Qwen-14B-Chat

At minimum, run openvino_tokenizers check against 2-3 models of the same type as the fixed tokenizer.

Common Pitfalls

1. Don't forget parse(): TransformersTokenizerPipelineParser.__init__ does NOT populate the pipeline — you must call parser.parse() first.

2. Handler return types: Map handlers must return either a single *Step instance or a list[*Step]. The parser wraps singles in lists internally.

3. Regex dialect: OV uses PCRE2 regex (via the C++ RegexSplit / RegexNormalization ops). HF's Rust tokenizers uses the fancy-regex crate. Key differences: Unicode property classes (\p{L}, \p{N}) are supported in both, but lookahead/lookbehind support may differ.

4. Byte-level encoding: ByteLevel pre-tokenizer in HF does TWO things: regex split AND byte→char mapping. In OV these are separate steps: RegexSplitStep + BytesToCharsStep. The BytesToCharsStep is later removed during BPETokenizationStep.finalize() which absorbs it into the vocab.

5. Step ordering matters: Pre-tokenization steps are applied sequentially. merge_regex_split_steps combines them into fewer ops for efficiency, but this changes the execution from sequential to parallel |-alternation.

Security

• NEVER run arbitrary code from tokenizer.json — only read data fields
• NEVER install packages — assume the environment is pre-configured
• NEVER modify model_id — pass it exactly as provided
• Validate that regex patterns from tokenizer.json are used only via the safe OV regex API (no eval(), no subprocess)