Gptqmodel Tokenizer Normalization

Diagnose and correct GPT-QModel tokenizer initialization, tokenization normalization, special-token handling, prompt rendering, and chat-template problems. Use when inference or evaluation quality suggests wrong token IDs or prompts, when a model needs tokenizer load kwargs or compatibility patches, or when tokenizer behavior is being changed in GPT-QModel; reusable corrective behavior must be implemented, tested, versioned, pushed, and submitted upstream to github.com/ModelCloud/Tokenicer.

modelcloud Updated
Picture from the README

File contents

modelcloud/gptqmodel commit 03a7211d14

Frequently asked questions

npx get-skill add modelcloud/gptqmodel-tokenizer-normalization