Vllm Tool Parsers

vLLM Tool Parsers — Navigation Map

This skill points to the right source file, template, or GH issue. The source code is authoritative — read it. Do not paraphrase from this skill when the actual file is available.

Where things live

Assume a local vllm-project/vllm checkout is accessible. Every reference below is relative to that repo root.

If the operator's question is "what does parser X do" — read vllm/tool_parsers/X_tool_parser.py. Don't rely on this skill's paraphrase.

Except for the 13 names on the unified-parser path, where that file is a stub of a few lines and the logic lives in vllm/parser/<model>.py:

_engine_ in the filename is not the marker. glm47_moe_tool_parser.py, kimi_k2_tool_parser.py, minimax_m2_tool_parser.py and mistral_tool_parser.py have ordinary names and are still stubs. The test is whether the file imports the adapter: grep -l "registered_adapters import" vllm/tool_parsers/*.py.

This is the same refactor described in vllm-reasoning-parsers — a single per-model parser now backs both the tool and reasoning adapters (RFC #32713, still formally OPEN and stale-bot-marked while the code ships). Practical consequence: a grammar change to vllm/parser/qwen3.py moves tool and reasoning behaviour at once — they are no longer independent surfaces for those models.

The CLI contract

Two flags, both required together for auto tool choice:

vllm serve <model> --enable-auto-tool-choice --tool-call-parser <name> [--chat-template <path>]

• --enable-auto-tool-choice alone → TypeError: --enable-auto-tool-choice requires --tool-call-parser (see cli_args.py).
• --tool-call-parser alone → legal. Parser still runs for tool_choice="required" and named, and on Responses API.
• No auto sentinel. Name a concrete parser.
• --tool-parser-plugin <path.py> → third-party file that calls @ToolParserManager.register_module("name").
• --reasoning-parser is independent but several tool parsers assume a </think> has closed — match them (see "Reasoning pairing" below).
• Chat template often matters. Each parser has a reference Jinja at examples/tool_chat_template_<family>.jinja. Wrong template → model never emits the sentinels the parser expects.

Parser → model family index

Use this to pick the CLI name. Then read the parser file and the matching Jinja for details — the wrapping tokens, streaming strategy, and quirks live there, not here.

gpt-oss/Harmony changed at v0.27.0 (PR #45560). json_object/json_schema response_format is now rewritten into a Harmony-aware structural_tag in HarmonyParser.adjust_request, so constrained decoding governs the whole generation instead of only the post-<|channel|>final<|message|> region. Without builtin tools the grammar validates tool name + arguments; with builtin tools it falls back to "some tool is called" only.

Don't trust this table to be complete — verify with:

grep -E "^\s+\"" vllm/tool_parsers/__init__.py    # lists registered names
ls examples/tool_chat_template_*.jinja            # lists shipped templates
ls vllm/tool_parsers/*_tool_parser.py              # lists source files

Framework contract (mental model)

Worth carrying as mental model, because it's spread across multiple files and easy to miss:

• ToolParser subclass implements extract_tool_calls (non-streaming, stateless) and extract_tool_calls_streaming (stateful, per-delta). See vllm/tool_parsers/abstract_tool_parser.py.
• Serving-layer invariants (guaranteed to the streaming method):
  • current_text == previous_text + delta_text
  • current_token_ids == previous_token_ids + delta_token_ids
  • Deltas may span multiple tokens.
• Four state fields the parser MUST maintain:
  • prev_tool_call_arr: list[dict] — serving reads [i]["arguments"] at stream end to flush the tail. If empty at end, finish_reason becomes stop not tool_calls.
  • current_tool_id: int — starts -1, increments per call.
  • current_tool_name_sent: bool — flip True once name flushed for current tool.
  • streamed_args_for_tool: list[str] — cumulative args already emitted per tool index. Append on every flush or the tail double-streams.
• Optional: adjust_request(request) — set skip_special_tokens=False, inject grammar, etc. supports_required_and_named: bool = True — flip False if the output shape breaks guided JSON.
• Return-value contract for streaming: None = "consumed, nothing to emit"; DeltaMessage(content=...) = pass-through; DeltaMessage(tool_calls=[DeltaToolCall(...)]) = tool progress.

For the parse_delta refactor see RFC #11522 (closed 2025-09-05) and its follow-on PRs #38755 (merged 2026-04-08), #39728 (merged 2026-04-13), #39446 (merged 2026-04-14). Align new parsers with the parse_delta shape rather than copying older HACKs.

Reasoning-parser pairing

Several tool parsers gate on a reasoning-end sentinel. Mismatched pair = tool parser sees reasoning text as content, misses sentinels, or emits from inside <think>. Table below is a pointer — verify by reading the tool-parser file for the adjust_request/is_reasoning_end interaction.

Sibling skill: vllm-reasoning-parsers — defer there for reasoning-side questions.

Diagnostic playbook

When a user reports a broken tool call, work down this list. Each step names where to look.

1. Both flags set? Check serve command for --enable-auto-tool-choice --tool-call-parser <name>.
2. Right parser for the model? Cross-check with the table above AND with vllm/tool_parsers/__init__.py registry.
3. Chat template matches? Inspect the actual template bytes — don't trust the filename. See step 7.
4. Reasoning parser paired? If the model has <think> / </seed:think> / harmony channels, --reasoning-parser must match. Check vllm/reasoning/ for the registry.
5. Streaming vs non-streaming? Grep the parser file: if extract_tool_calls_streaming returns None unconditionally or raises NotImplementedError, streaming isn't supported (e.g. phi4_mini_json, openai).
6. finish_reason = stop? Then prev_tool_call_arr is empty at stream end — parser failed to populate it. Enable vLLM debug logs and trace.
7. Raw model output vs what parser sees. Bypass the parser: call /v1/completions (no tool parser) with the same prompt. Dump raw bytes:

   curl -sS $VLLM/v1/completions -H 'content-type: application/json' \
     -d '{"model":"...","prompt":"...","max_tokens":200}' \
     | python3 -c "import sys,json,unicodedata; t=json.load(sys.stdin)['choices'][0]['text']; \
         [print(hex(ord(ch)), unicodedata.name(ch,'?'), repr(ch)) for ch in t if ord(ch)>0x7E][:40]"
   

   This distinguishes "model not emitting sentinels" (template/training problem) from "parser not matching sentinels" (parser bug).
8. vLLM version vs known bugs. See "Bug archaeology" below.
9. Custom template? Diff the deployed Jinja against examples/tool_chat_template_<family>.jinja. Character-level. Full-width vs ASCII Unicode gotchas bite here.

Bug archaeology (don't trust static lists — search)

Tool-parser bugs churn quickly. Teach yourself the pattern:

# Open bugs affecting a specific parser
gh search issues --repo vllm-project/vllm "<parser-name> tool" --state open \
  --json title,url,state,updatedAt --limit 20

# Recently merged fix PRs
gh search prs --repo vllm-project/vllm "<parser-name>" --state merged \
  --json title,url,mergedAt --limit 20

# Streaming-specific
gh search issues --repo vllm-project/vllm "tool parser streaming <parser-name>" \
  --json title,url,state,updatedAt --limit 20

# Broad sweep if parser name unknown
gh search issues --repo vllm-project/vllm "extract_tool_calls_streaming" --limit 30

On finding a referenced issue/PR, read it directly (gh issue view N --repo vllm-project/vllm --comments). Never paraphrase from a cached memory — the fix may have landed since.

Umbrella RFC to know about: #11522 — "Refactor tool parsers to eliminate coding errors". Tracks the parse_delta refactor (PRs #39446, #39728, #38755). Any new parser work should align with it.

In-tree HACKs to recognize

Helpful to know what to look for when reading a parser. Grep the codebase for these:

grep -rn "HACK" vllm/tool_parsers/
grep -rn "TODO" vllm/tool_parsers/
grep -rn "prev_tool_call_arr = \[{\"arguments\": {}}\]" vllm/tool_parsers/

The prev_tool_call_arr = [{"arguments": {}}] plant is the classic "force finish_reason=tool_calls" workaround. At v0.27.0 exactly five parsers carry it: pythonic, llama4_pythonic, olmo3, lfm2, minicpm5xml — the pythonic family plus the two that reuse its flush shape. (mistral does not; it populates prev_tool_call_arr properly.) When writing a new parser, prefer the parse_delta shape (RFC #11522).

Writing a custom parser

Copy the existing parser closest in format. Don't reinvent.

# Match the target shape to an existing parser
grep -lE "<your_sentinel>" vllm/tool_parsers/   # sometimes the sentinel itself is in the code
ls vllm/tool_parsers/                            # scan names by format

For the skeleton + detailed checklist see references/custom-parser-plugin.md. Key points:

• Register with @ToolParserManager.register_module(["name"]) (decorator path is lazy — plugin loader imports the file).
• Launch: --tool-parser-plugin /abs/path/file.py --tool-call-parser name.
• Closest starting point for most models: vllm/tool_parsers/pythonic_tool_parser.py (kwargs Python syntax), hermes_tool_parser.py (JSON-in-tags), or step3p5_tool_parser.py (expat streaming XML — qwen3xml_tool_parser.py was deleted in the unified-engine refactor).
• Must honor the four state fields above. Read vllm/tool_parsers/abstract_tool_parser.py for exact signatures.
• Read tests/tool_parsers/common_tests.py — reviewers expect this harness.

Before upstreaming, read AGENTS.md at the repo root for the duplicate-work + PR-description policy.

References in this skill

Compact supplementary maps. Each points back at the source rather than duplicating it.

• references/parser-index.md — one-line-per-parser index with file path, wrapping tokens at a glance, and the things unique to that parser that aren't obvious from the filename (e.g. "full-width ｜ U+FF5C sentinels", "non-streaming only", "schema-aware type coercion").
• references/streaming-pitfalls.md — the things that bite across parsers: full-width pipes, special-token stripping, empty-args stalls, HACKs, diagnostic flow. Points at chat_completion/serving.py for the flush contract.
• references/custom-parser-plugin.md — plugin scaffolding checklist with file:line anchors into the base class and the canonical examples.
• references/sources.md — verification log of the external GitHub issues/PRs/source files cited by this skill, each with a Last verified date. Consult before re-citing a claim; re-probe if the entry is stale (>90 days).

(Older references/json-family.md, pythonic-xml-family.md, known-bugs.md removed — they duplicated source and rotted. Use Grep/gh search instead.)

Last verified: 2026-08-11 against vLLM v0.27.0. See references/sources.md for per-reference probe details and timestamps.