Transformers Config Tokenizers Expert

Transformers config + tokenizers expert

Target: engineers writing a preflight tool (or a vLLM/sglang operator) that must decide, before handing a HuggingFace snapshot to an inference engine, which files win, which tokens are structural, and which class will actually instantiate.

Almost every major 2026 release has shipped with drift between tokenizer_config.json, generation_config.json, config.json, and the Rust-backed tokenizer state. The skill exists so a preflight tool can answer that drift authoritatively — not guess.

Stance

• Cite, don't paraphrase. Every load-bearing claim has a file:line or URL citation in references/. Point at the source.
• Version-gate. Transformers v5 (GA 2026-01-26) renamed the tokenizer classes and changed serialization shapes. Pre-5.0 and post-5.0 diverge — check transformers.__version__ before claiming.
• Rust is truth. For any model with tokenizer.json, the authoritative added-token state is tokenizer.backend_tokenizer.get_added_tokens_decoder(). Python-side all_special_ids / special_tokens_map / added_tokens_decoder are views; treat them as such.
• Engines disagree. vLLM and sglang both union-merge generation_config.eos_token_id, but apply it through different pipelines (see engine-knobs.md). Predict per engine, not in the abstract.

Triage: symptom → layer → reference

Use this table first. Deep dives live in references/.

The precedence cheat sheet (memorize)

Five sources exist for "is this token structural?" They disagree. Reach for them in this order when writing preflight code:

1. tokenizer.backend_tokenizer.get_added_tokens_decoder() — Rust truth. dict[int, AddedToken]. Every added token, with special flag, lstrip/rstrip/normalized attrs. Source: tokenization_utils_tokenizers.py:488-495 (v5), passthrough. Only available for TokenizersBackend.
2. tokenizer.added_tokens_decoder — Python mirror. For TokenizersBackend it's a passthrough to #1. For PythonBackend it's deserialized from tokenizer_config.json["added_tokens_decoder"].
3. tokenizer.all_special_tokens / all_special_ids — the narrow union of SEVEN named role slots (SPECIAL_TOKENS_ATTRIBUTES at tokenization_utils_base.py:1040-1047) + extra_special_tokens. Does not include any added token registered with special=False. This is why DeepSeek's reserved-token slabs and GLM-5.1's <\|user\|>-as-turn-primer are invisible.
4. tokenizer.special_tokens_map — only the seven role slots as dict[str, str]. No extras, no added_tokens. Legacy shape.
5. tokenizer.extra_special_tokens — list internally (self._extra_special_tokens = [] at tokenization_utils_base.py:1074). v5.0.0rc0 serialized this as a list into tokenizer_config.json, crashing <5.0 readers that call .keys(). GLM-5.1 ships extra_special_tokens as a list; GLM-4.6 ships it as {} (dict). See precedence-rules.md#extra-special-tokens-shape.

Preflight rule of thumb. For any structural question beyond "is this a named role slot", go to #1. If #1 is unavailable (no tokenizer.json, i.e. Kimi via TikTokenTokenizer), fall through to #2 from tokenizer_config.json["added_tokens_decoder"], and cross-ref against generation_config.json.

Full table with file:line per backend: references/precedence-rules.md.

Config-file precedence (memorize)

For the "which EOS wins" question:

Consequence: a single-int eos_token in tokenizer_config.json paired with a three-ID list in generation_config.json is fine for engines (they union) but ambiguous for any tool that only reads the tokenizer. Preflight must read both and diff.

Full catalogue per file: references/config-files.md.

Tokenizer-class cross-reference (2026)

Which class actually instantiates for major lab repos. Verified against each repo's tokenizer_config.json.

Full taxonomy + auto_map mechanics: references/tokenizer-classes.md.

Chat-template Jinja rendering contract

Environment built at transformers/utils/chat_template_utils.py:234:

jinja_env = ImmutableSandboxedEnvironment(
    trim_blocks=True, lstrip_blocks=True,
    extensions=[AssistantTracker, jinja2.ext.loopcontrols]
)
jinja_env.filters["tojson"] = tojson                     # ensure_ascii=False default
jinja_env.globals["raise_exception"] = raise_exception   # throws TemplateError
jinja_env.globals["strftime_now"] = strftime_now         # LOCAL TZ, not UTC

Four gotchas operators hit:

1. tojson defaults to ensure_ascii=False — stdlib Jinja's default is True. Templates that dump CJK/emoji tool schemas rely on this override. A preflight tool that renders in a naive Jinja env will produce HTML-escaped output the model never trained on.
2. strftime_now uses local time. Llama-3.1/3.2 templates inject a date header; the host's timezone determines the value. A container running in UTC produces different prompts than a laptop in Europe.
3. ImmutableSandboxedEnvironment blocks mutation. Templates cannot .pop() messages or write to passed objects. Workarounds copy into locals.
4. loopcontrols enables {% break %} and {% continue %}. Some templates depend on these; a stripped-down renderer missing the extension raises TemplateSyntaxError.

add_generation_prompt semantics, continue_final_message, apply_chat_template resolution order, AssistantTracker offsets: references/chat-template-contract.md.

Engine knob precedence (vLLM + sglang)

Short form:

• vLLM chat_template_kwargs: CLI --default-chat-template-kwargs → OpenAIServingChat.__init__ default → _prepare_extra_chat_template_kwargs merges with dict-union (request wins) → safe_apply_chat_template → resolve_chat_template_kwargs allowlist filter at vllm/renderers/hf.py:352-377 → tokenizer.apply_chat_template(**resolved). Allowlist fix PR #27622 shipped in v0.11.1 (2025-11-18). Pre-v0.11.1 silently dropped kwargs for tokenizers whose apply_chat_template uses **kwargs (Kimi K2).
• sglang chat_template_kwargs: literal dict update at serving_chat.py:524-527. No allowlist. Any key reaches apply_chat_template. Closer to pre-27622 vLLM.
• vLLM trust_request_chat_template: default False. Rejects per-request chat_template or chat_template_kwargs unless set True. Enforced at engine/serving.py:415-425.
• sglang no equivalent: request kwargs always accepted; only three sites hardcode overrides to skip_special_tokens=False (gpt-oss/gemma4 models, request.tools present, mistral reasoning_effort).
• adjust_request (vLLM): runs at render/serving.py:372-383, reasoning parser first then tool parser. Can mutate tools, stop, structured_outputs, response_format before to_sampling_params.
• sglang has no adjust_request analog. The three hardcoded skip_special_tokens=False overrides at serving_chat.py:306/315/397 are the equivalent.
• Stop-token merge:
  • vLLM: update_from_generation_config at sampling_params.py:540-560 appends generation_config.eos_token_id list to stop_token_ids unless ignore_eos=True.
  • sglang: model_config._get_hf_eos_token_id at model_config.py:580-598 unions hf_config.eos_token_id and hf_generation_config.eos_token_id into Set[int].
• Incremental detokenizer word boundaries: vLLM has fast (DecodeStream from tokenizers) and slow (detokenize_incrementally with prefix_offset/read_offset diff + U+FFFD guard) paths at vllm/v1/engine/detokenizer.py and vllm/tokenizers/detokenizer_utils.py:98-167. sglang uses DetokenizerManager subprocess with four-offset DecodeStatus at sglang/srt/managers/detokenizer_manager.py:57-63. sglang #22510 was a serving_chat.py double-slice bug (fixed PR #22549, not the detokenizer — despite skip_special_tokens=False being a red herring in the initial report).

Deep dive with file:line per knob: references/engine-knobs.md.

Hall of shame (verified 2026)

Pre-loaded real incidents. Each entry in references/hall-of-shame.md has the exact file(s), token IDs, and — where known — the bead ID or commit SHA. Summary:

• GLM-5.1 — three-ID EOS [154820, 154827, 154829] in generation_config.json. IDs 154827/154829 are <|user|> / <|observation|> turn primers. Engines unioning this list stop on turn boundaries; skip_special_tokens=False leaks them into output. extra_special_tokens as list, not dict. TokenizersBackend class name — fails import on transformers <5.0.
• GLM-5.1-FP8 orphan-commit trap — patch SHA 6ad52ee not reachable from refs/heads/main (a92f8155). Users assume fix is live; it isn't. Verification requires checking /refs on HF API.
• Gemma-4-26B-A4B-it — multi-ID EOS [1, 106, 50]. added_tokens_decoder absent from tokenizer_config.json (lives in LFS tokenizer.json). Separate chat_template.jinja not auto-loaded by transformers 5.5.0 (issue #45205). Scalar-null serialization bug in format_argument macro renders None not null.
• Kimi-K2.6 half-fix — config.json + generation_config.json flipped EOS to 163586 (<|im_end|>); tokenizer_config.json kept [EOS] (163585). vLLM reads tokenizer_config, sglang reads generation_config. Different engines stop on different tokens. No tokenizer.json; tiktoken package required.
• Kimi-K2.6 nested-config trap — quantization_config lives at config["text_config"]["quantization_config"], NOT top-level. Top has only dtype: bfloat16 and an empty/absent quant block. A reader that grabs config["quantization_config"] returns {} and concludes "BF16, no quantization" — wrong. Reality: compressed-tensors, num_bits: 4, group_size: 32, format: pack-quantized (W4A16 routed-MoE INT4 with BF16 carve-outs for lm_head, self_attn.*, shared_experts.*, dense MLP). Total checkpoint 595 GB ≠ ~1 TB BF16. Always walk nested keys — text_config, vision_config, audio_config, language_config are common multimodal/MoE homes. Same trap on K2.5 (same nesting), Llama-4 vision configs, GLM-4V, Qwen3-VL.
• Qwen3-0.6B — <|im_end|> is simultaneously turn terminator AND EOS. Qwen3.5-Base flips EOS to <|endoftext|> — preflight hardcoding <|im_end|> emits runaway completions on base variants.
• DeepSeek-V3 — added tokens live only in tokenizer.json (7.85 MB LFS). tokenizer_config.json has no added_tokens_decoder. <think>/</think> on R1 live only in chat_template.jinja, not as added tokens.
• Phi-4 inversion — EOS <|im_end|>, BOS <|endoftext|>. Opposite of Qwen-Base. Don't regex on string.

Full incidents with citations: references/hall-of-shame.md.

Drop-in snippets

references/snippets.py — copy-paste Python for preflight init-time questions:

Reference map

• references/config-files.md — catalogue per file, drift matrix
• references/tokenizer-classes.md — v5 taxonomy, auto_map, aliases
• references/precedence-rules.md — five-source discovery w/ file:line
• references/chat-template-contract.md — Jinja env, globals, add_generation_prompt
• references/engine-knobs.md — vLLM + sglang tokenizer-adjacent flags
• references/hall-of-shame.md — verified 2026 incidents
• references/snippets.py — drop-in preflight Python
• references/sources.md — dated external references (freshen target)