How do I install the Vllm Input Modalities skill?

Run npx skillmds@latest add air-gapped/vllm-input-modalities in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Vllm Input Modalities skill do?

vLLM non-chat inference surfaces — text embeddings (`/v1/embeddings`, `/v2/embed`), reranking/scoring (`/rerank`, `/score`), speech-to-text (`/v1/audio/transcriptions`, `/v1/audio/translations`), document OCR via VLMs. Covers 2026 `--runner pooling` (replacing `--task embed`), v0.20 deprecations (`score`→`classify`, multitask pooling, `encode`→`token_embed`+`token_classify`), Matryoshka/MRL, ColBERT/ColPali/ColQwen late-interaction MaxSim, Cohere v2 `/v2/embed`, Jina v3/v4/v5 quirks, cross-encoder score templates, Whisper large-v3-turbo quants, DeepSeek-OCR recipe (NGramPerReqLogitsProcessor, no prefix cache, GUNDAM mode). It is listed under AI & ML on SkillMD.

Is Vllm Input Modalities safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Vllm Input Modalities?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Vllm Input Modalities free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Vllm Input Modalities?

air-gapped (@air-gapped) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Vllm Input Modalities by air-gapped

Name: Vllm Input Modalities
Author: air-gapped

vLLM — embeddings, reranking, speech-to-text, OCR

Target audience: operators who need vLLM's non-chat-completion surfaces. Four capabilities bundled here because they share operator-facing concepts (--runner flag, pooling configuration, scoring API, multimodal preprocessing) even though two run on the pooling runner (embedding, reranking) and two run on the generate runner (STT, OCR).

The mental model — one flag rules the surface

vLLM decides what a model does from the combination of three flags:

--runner {auto|generate|pooling|draft}      # what kind of workload
--convert {auto|none|embed|classify}        # adapt a generative LM to a pooler
--pooler-config '{...}'                     # override pool type, dimensions, etc.

The pair (runner, convert) has replaced the old --task {generate|embed| score|classify|reward|...} flag. --task is gone, not deprecated — there is no task field in ModelConfig and no --task in vllm/engine/arg_utils.py at v0.27.0, so passing it is an unrecognized-argument error. Canonical today:

Workload	Command	Runner	Notes
Chat / completion	`vllm serve <model>`	`generate` (auto)	default
Embedding	`vllm serve <model> --runner pooling`	`pooling`	auto-detects CLS/LAST/MEAN from config
Embedding from a causal LM	`vllm serve <model> --runner pooling --convert embed`	`pooling`	adapts `*ForCausalLM` checkpoints
Classification	`vllm serve <model> --runner pooling --convert classify`	`pooling`	also how `score`/`rerank` comes online
Speech-to-text	`vllm serve <model>`	`generate`	works on any `SupportsTranscription` model
OCR (VLM generate)	`vllm serve <model>`	`generate`	standard chat-completion + image input

Scoring API is automatic. There is no --enable-scoring-api. The /score + /rerank endpoints light up whenever the loaded model's Pooler.get_supported_tasks() includes classify (with num_labels==1), embed, or token_embed (late-interaction). Nothing for the operator to toggle.

Quick-answer router

Question class	File
"Which pooling type? Matryoshka? `/v2/embed`? BGE-M3, Qwen3, Jina?"	`references/embedding.md`
"Cross-encoder vs ColBERT? Qwen3-Reranker? BGE-reranker? Score templates?"	`references/reranking.md`
"Whisper-turbo? Voxtral? Qwen3-ASR? Chunking? Quants?"	`references/stt.md`
"DeepSeek-OCR recipe? dots-OCR? VLM document parsing?"	`references/ocr.md`
"Is `--task embed` gone? What replaces `encode`?"	`references/runner-flags.md`

scripts/probe-endpoint.sh checks a running vLLM whether it exposes /v1/embeddings, /rerank, /v1/audio/transcriptions, etc., so an operator can confirm the right endpoints are live before pointing a client at it.

Operator cheat sheet — the common cases inline

Embedding

# Qwen3-Embedding (causal LM, last-token pooling — auto-detected)
vllm serve Qwen/Qwen3-Embedding-0.6B --runner pooling

# BGE-M3 (XLM-Roberta, CLS pooling — native embedding model)
vllm serve BAAI/bge-m3 --runner pooling

# Jina v3 (needs trust-remote-code; only text-matching LoRA is merged)
vllm serve jinaai/jina-embeddings-v3 --runner pooling --trust-remote-code

# Jina v4 — use the pre-merged retrieval variant
vllm serve jinaai/jina-embeddings-v4-vllm-retrieval --runner pooling \
  --pooler-config '{"pooling_type":"ALL"}' --dtype float16
# Normalization happens client-side (vector is multi-vector per token).

# Mean-pool override (Sentence-Transformers config is broken for this model)
vllm serve ssmits/Qwen2-7B-Instruct-embed-base --runner pooling \
  --pooler-config '{"pooling_type":"MEAN"}'

Client request format is standard OpenAI: client.embeddings.create(...).

Matryoshka dimensions. Gated on is_matryoshka: true in the model's config.json (or matryoshka_dimensions). If the config is missing it, force-enable:

--hf-overrides '{"is_matryoshka": true}'
# or pin specific dimensions:
--hf-overrides '{"matryoshka_dimensions":[256,512,768]}'

Request-side: client.embeddings.create(model=..., input=..., dimensions=512). Passing dimensions to a non-MRL model (BGE-M3, older BGE) returns a 400 by design — not a bug.

dimensions above the model's hidden_size is rejected as of v0.24.0 (#46313). Before that, an MRL model with no explicit matryoshka_dimensions list validated only dimensions >= 1 and then sliced [..., :d] — so an oversized value silently returned a hidden_size-length vector instead of erroring. If a client has been over-asking, it was never getting the width it requested; upgrading turns that into a visible ValueError. Pin the list explicitly via --hf-overrides to make the valid set unambiguous.

/v2/embed (Cohere v2 compat) adds input_type prompt prefixing, output_dimension (server-side MRL), truncate=END|START|NONE, and embedding_types=["float","binary","ubinary","base64"]. Use it when a client expects Cohere v2's shape.

Reranking / scoring

Three serving modes; same endpoints, picked automatically:

# Cross-encoder (classify, num_labels==1)
vllm serve BAAI/bge-reranker-v2-m3 --runner pooling

# Cross-encoder with instruction-aware score template
vllm serve Qwen/Qwen3-Reranker-0.6B --runner pooling --convert classify \
  --chat-template examples/templates/qwen3_reranker.jinja \
  --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],
                    "classifier_from_token":["no","yes"],
                    "is_original_qwen3_reranker":true}'

# Late-interaction (ColBERT family) — MaxSim over token embeddings
vllm serve jinaai/jina-colbert-v2 --runner pooling --trust-remote-code

# Multimodal reranker (ColPali / ColQwen)
vllm serve vidore/colpali-v1.3-hf --runner pooling

Client:

# /rerank (Cohere + Jina compat)
resp = requests.post("http://localhost:8000/rerank", json={
    "query": "what is vLLM",
    "documents": ["text 1", "text 2"],
    "top_n": 3,
    "max_tokens_per_doc": 512,  # added v0.20.0 (PR #38827)
})

# /score (bi-encoder cosine, or cross-encoder logit)
resp = requests.post("http://localhost:8000/score", json={
    "text_1": ["query"],
    "text_2": ["doc A", "doc B"],
})

Three score_types served through the same routes:

Score type	Mechanism	Models
cross-encoder	joint query+doc forward → single logit	BGE-reranker-v2-m3/gemma, Qwen3-Reranker, mxbai-rerank-v2, nvidia/llama-nemotron-rerank
late-interaction	per-token embeddings + MaxSim	ColBERT, ColModernBERT, jina-colbert-v2, ColPali, ColQwen3/3.5, ColModernVBert
bi-encoder	cosine over `/embeddings`	any embedding model (auto)

jinaai/jina-reranker-v3 is listwise ("last but not late interaction") — JinaForRanking, not MaxSim.

Speech-to-text

# Whisper large-v3-turbo (base)
vllm serve openai/whisper-large-v3-turbo

# Red Hat production quants (fit on smaller cards, validated)
vllm serve RedHatAI/whisper-large-v3-turbo-FP8-dynamic
vllm serve RedHatAI/whisper-large-v3-turbo-quantized.w8a8
vllm serve RedHatAI/whisper-large-v3-turbo-quantized.w4a16

# Voxtral (Mistral)
vllm serve mistralai/Voxtral-Mini-3B-2507

Client:

curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "file=@audio.wav" \
  -F "model=openai/whisper-large-v3-turbo" \
  -F "language=en"

Chunking >30 s audio is server-side (energy-aware split at min_energy_split_window_size). Beam-search transcription arrived in v0.18.

OOM on 24 GB with Whisper (issue #15216) is a known sharp edge — Whisper allocates aggressively for its encoder KV, despite the 1.6 GB checkpoint. Production path is one of the RedHatAI quants above, or raising --gpu-memory-utilization past 0.9 with eager mode if memory is truly tight.

OCR (DeepSeek-OCR)

Canonical recipe from docs.vllm.ai/projects/recipes:

vllm serve deepseek-ai/DeepSeek-OCR \
  --logits-processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \
  --no-enable-prefix-caching \
  --mm-processor-cache-gb 0

Three non-obvious flags:

NGramPerReqLogitsProcessor is required — without it, table-token generation degrades. Enforces whitelist_token_ids={128821,128822}, ngram_size=30, window_size=90.
Disable prefix caching. OCR per-request inputs don't share prefixes; the cache bookkeeping is pure overhead.
--mm-processor-cache-gb 0 — the multimodal processor cache isn't useful for one-off document images.

DeepSeek reports ~2500 tok/s per A100-40 GB, ~200 k pages/day per GPU. Mode is hard-coded to GUNDAM (base=1024, image=640, crop=True); Tiny/Small/Base/ Large aren't exposed via env vars yet (tracked issue, as of early 2026).

Invocation is still plain /v1/chat/completions with image URLs — there is no dedicated /ocr endpoint.

Top pitfalls

--task embed is dead, not deprecated. The flag no longer exists — passing it fails argument parsing. Use --runner pooling. The score and encode pooling task names are removed too and raise VLLMValidationError (vllm/tasks.py): use --convert classify on a num_labels==1 model to light up /score + /rerank, and token_embed / token_classify in place of encode.
Pooling runs on PIECEWISE CUDA graphs, not full graphs. That's deliberate (pooling models have variable-shape outputs). Don't force --enforce-eager for production as older cheat sheets suggest — you lose the piecewise graph win without gaining anything.
Jina v4 base checkpoint is not vLLM-compatible. Use jinaai/jina-embeddings-v4-vllm-retrieval (pre-merged retrieval adapter). Serve with --pooler-config '{"pooling_type":"ALL"}' --dtype float16 and normalize client-side — output is multi-vector per token.
Matryoshka without config. If a model documents MRL support but config.json lacks is_matryoshka / matryoshka_dimensions, the server returns 400 for any dimensions param. Fix: --hf-overrides '{"is_matryoshka":true}' at serve time. Don't confuse with BGE-M3, which genuinely doesn't support MRL.
Qwen3-Reranker needs a score template AND hf-overrides. It's an instruction-tuned causal LM masquerading as a cross-encoder — skipping any of the three extras (see the reranking cheat-sheet command above) gives random-looking scores, not errors. Full recipe: references/reranking.md §2. Second cause of the same symptom, below v0.26.0: #48901 — any LAST-pooling model returns wrong scores once a query+doc pair is long enough to be chunk-prefilled, under torch.compile (the default). Rule out the version before re-checking the overrides.
DeepSeek-OCR with prefix caching on. It doesn't crash — it just wastes time and memory. Same for --mm-processor-cache-gb > 0 for pure OCR traffic. Both defaults are wrong for this workload.
Whisper OOM on 24 GB. Not a bug. Use a Red Hat quant, or accept that large-v3 / large-v3-turbo wants ≥32 GB for comfortable batch sizes.
Late-interaction kernel regression sniff test. ColBERT / ColPali throughput jumped ~14% in v0.17–0.19 from MaxSim optimisations. If those models feel slow, check --enable-flash-late-interaction (default true) wasn't disabled by an old config.

Landed in v0.20.0 (released 2026-04-27) — verify your deployment

The deprecations previously flagged as "scheduled for v0.20" have now shipped. Callouts from the v0.20.0 release notes (Breaking Changes + API sections):

logit_bias / logit_scale → logit_mean / logit_sigma in PoolerConfig — explicit breaking change, PR #39530. Old names still accepted with deprecation warning.
Async scheduling default OFF for pooling models (PR #39592) — explicit breaking change. Pooling throughput should be marginally lower but stability improves; re-enable case-by-case if you measured a win on v0.19.
--task flag — still accepted with deprecation warning; --runner + --convert is canonical.
score pooling task — replaced by classify + num_labels==1.
Pooling multitask — pick a task explicitly via PoolerConfig(task=...) or --pooler-config.task <task>; automatic multitasking is gone.
encode task — split into token_embed and token_classify.
normalize in PoolingParams — removed; use use_activation.

Two performance wins also landed in v0.20.0 for pooling:

#38559 — mean-pooling optimisation via index_add (+5.9% on mean-pool models).
#39113 — redundant-sync removal for pooling (+3.7% throughput).

Also landed: jina-reranker-v3 (#38800), Jina Embeddings v5 (#39575), max_tokens_per_doc in /rerank (#38827), Generative Scoring (#34539), ASR multi-chunk spacing fix (#39116).

v0.21.0 → v0.25.1

The v0.20.0 migration above is still the canonical runner surface — nothing in v0.21–v0.25 changed --runner / --convert / PoolerConfig. But two request-validation changes in v0.24.0 will turn requests that used to succeed into 400s, so they are the ones to check before upgrading.

Two silent-success → hard-error changes (v0.24.0):

#46313 — matryoshka dimensions above hidden_size is now rejected. For an MRL model with no explicit matryoshka_dimensions list, the old code only checked dimensions >= 1 and then sliced [..., :d], so an oversized request silently returned a hidden_size-length vector. It now raises. A client that has been asking for e.g. dimensions=2048 against a 1024-hidden model was already getting 1024 floats back and will now get an error instead — the error is the fix, but it surfaces at upgrade time.
#46119 — rerank top_n must be non-negative. top_n=-1 was silently treated as top_n=0. top_n=0 still means "return all results", and values larger than the document count are still accepted.

New capability worth adopting (v0.24.0):

#45173 — /v1/embeddings accepts message-shaped input and chat_template_kwargs. Previously message-shaped input to /v1/embeddings was rejected at validation and chat_template_kwargs never reached the renderer; only the top-level messages extension worked. This is the supported path for instruction-style embedding prompts.
#45640 Cohere /v2/embed input-exclusivity validation; #44999 / #45210 ColBERT AutoWeightsLoader plus a query/document embedding io-processor.

Not applicable despite the release-note wording: v0.22.0 #43260 "add truncation side to OpenAI endpoints" covers /v1/completions and /v1/chat/completions only — it does not add truncation_side to /v1/embeddings.

Perf / internals, no action required: #41163 AllPool.forward +51% and #41433 GPU↔CPU pooling sync elimination (v0.21.0); #42267 pooling offline API split into PoolingOfflineMixin, #42370/#42274 Speech-to-Text entrypoint + test consolidation (refactor, no endpoint change) (v0.22.0); #44593 proper pooling exceptions, #44410 LoRA-adapter-name pooling fix (v0.23.0); #44612 ASR CPU preprocessing 2.5× faster (v0.24.0); #46762 realtime embeddings under Model Runner V2 and #47071/#47437 pooled-Whisper sliding-window KV sizing — the latter had been over-reserving encoder KV blocks by roughly block_pool_size× (v0.25.0).

New architectures since v0.21.0: Qianfan-OCR (#40136, v0.21.0), Unlimited OCR (#46564 + Triton R-SWA backend #47102, v0.25.0), MOSS-Transcribe-Diarize (#47729, v0.25.0 — long-form transcription with timestamped speaker labels, Whisper-style encoder into a Qwen3 decoder), LLaVA-OneVision-2 (#44785). See references/ocr.md §2 and references/stt.md.

Ecosystem removals that can strand a deployment: v0.25.0 deleted PagedAttention entirely (#47361); v0.24.0 deprecated Transformers v4 support (#45161) and removed several model families outright (ERNIE, Xverse, Dots1, Bamba, Mono-InternVL); v0.25.0 removed Baichuan, Aquila, Grok, Tarsier/Tarsier2, AyaVision/MusicFlamingo, Mantis. None are pooling or STT models, but check before pinning a newer image for an unrelated reason.

v0.26.0 + v0.27.0 (baseline v0.27.0, released 2026-08-10)

Latest release is v0.27.1 (2026-08-11), a one-change patch — "quantized DSpark Markov heads" (#50424) — with nothing on this skill's surface. The claims below were verified against the v0.27.0 tag.

A pooling correctness bug, fixed in v0.26.0 — the highest-value item on this page. #48901: LAST-pooling models (the PR names Qwen/Qwen3-Reranker-0.6B) returned wrong relevance scores whenever a query+document pair was long enough for its prefill to be split into chunks, but only under torch.compile (the default). A pair scoring ~0.83 unchunked collapsed to ~0.01–0.36 and varied run to run. --enforce-eager was always correct and the offline batch path happened not to chunk, which is why it hid so easily. Any ranking a pre-v0.26.0 engine produced on long documents is suspect — this is a re-score, not just an upgrade. references/reranking.md §2.

Model Runner V2 reached pooling — but pooling still defaults to V1. v0.26.0/v0.27.0 landed encoder-only attention (#49331), sequence embed/classify pooling (#48791), token_classify (#50293), token_embed (#50574) and BGE-M3's embed&token_classify (#50661) on MRV2. The default-enable PR (#48290) is still open, so nothing changes unless VLLM_USE_V2_MODEL_RUNNER=1 is set. Details and the escape hatch: references/runner-flags.md §11.

STT request surface moved forward (v0.27.0): diarized_json response format (#48543), cumulative long-form chunk timestamps (#41131), extra sampling params on the translation API (#45839), MOSS-TD max audio duration raised to 90 minutes (#49403). The STT entrypoint package also moved on disk — references/stt.md §2.

New pooling architectures: jina-embeddings-v5-text-nano (#50688, v0.27.0 — EuroBERT encoder backbone under the existing JinaEmbeddingsV5Model); BertForMaskedLM (#48463), RobertaForTokenClassification / XLMRobertaForTokenClassification (#47991), LongCat-Flash-Lite n-gram embedding (#47857), all v0.26.0.

More removals, none on this surface: TeleChat (#47989), Persimmon and Fuyu (#48096) in v0.26.0; Plamo2 (#49729) and Ouro (#49786) in v0.27.0. Unrelated but image-bump relevant: max_num_partial_prefills and max_long_partial_prefills were removed in v0.27.0 (#49244).

Paired skills

vllm-configuration → environment variables, cache paths, telemetry opt-out.
vllm-observability → metrics exposition, Prometheus endpoints.
vllm-nvidia-hardware → SM-level platform support for pooling + FP8 paths.

Source and refresh policy

First-party: vLLM docs at https://docs.vllm.ai/en/stable/models/pooling_models/ (README + embed / scoring / token_embed / specific_models subpages), and docs/contributing/model/transcription.md in the repo.
Production STT canonical reference: Red Hat Developer blog for Whisper + RHAIIS (link in references/stt.md).
DeepSeek-OCR canonical reference: vLLM recipes page (link in references/ocr.md).
Refresh triggers: any v0.28+ release, MRV2 becoming the pooling default (#48290), a new Jina embeddings major version, or a new native-multimodal reranker shipping. Note that three passes running have found the runner surface quiet while request validation and pooling correctness moved underneath it — grep release bodies for pooling|rerank|embedding|matryoshka|top_n, not just for runner flags.
External-ref audit log: references/sources.md.

Last verified: 2026-08-11 (against vLLM v0.26.0 + v0.27.0 release notes, the PRs behind each pooling/rerank/STT/OCR hit, and the v0.27.0 source tree for the runner flags, task enums and MRV2 pooling gate). Headline: #48901 made pre-v0.26.0 LAST-pooling scores untrustworthy on chunk-prefilled pairs, and --task is now absent from the CLI entirely rather than deprecated.

Vllm Input Modalities

File contents

vLLM — embeddings, reranking, speech-to-text, OCR

The mental model — one flag rules the surface

Quick-answer router

Operator cheat sheet — the common cases inline

Embedding

Reranking / scoring

Speech-to-text

OCR (DeepSeek-OCR)

Top pitfalls

Landed in v0.20.0 (released 2026-04-27) — verify your deployment

v0.21.0 → v0.25.1

v0.26.0 + v0.27.0 (baseline v0.27.0, released 2026-08-10)

Paired skills

Source and refresh policy

Frequently asked questions

Vllm Input Modalities

File contents

vLLM — embeddings, reranking, speech-to-text, OCR

The mental model — one flag rules the surface

Quick-answer router

Operator cheat sheet — the common cases inline

Embedding

Reranking / scoring

Speech-to-text

OCR (DeepSeek-OCR)

Top pitfalls

Landed in v0.20.0 (released 2026-04-27) — verify your deployment

v0.21.0 → v0.25.1

v0.26.0 + v0.27.0 (baseline v0.27.0, released 2026-08-10)

Paired skills

Source and refresh policy

Related skills

Frequently asked questions

How do I install the Vllm Input Modalities skill?

What does the Vllm Input Modalities skill do?

Is Vllm Input Modalities safe to use?

Which AI agents work with Vllm Input Modalities?

Is Vllm Input Modalities free to use?

Who published Vllm Input Modalities?