Qwen38 Vllm Rtx Pro 6000

Serving Qwen3.8-27B on an RTX PRO 6000 Blackwell with vLLM

Stand up a production OpenAI-compatible endpoint on one Blackwell card: NVFP4 weights, built-in speculative decoding, working tool calls, vision, and an optional 1M-token context window.

Every number in this skill was measured on real hardware, not estimated. Everything here is plain vLLM + systemd — no agent framework required. Any client that speaks the OpenAI API (LangChain, LlamaIndex, Open WebUI, Cline, Continue, a raw curl) works against the result.

0. Who this is for

You have:

• An NVIDIA RTX PRO 6000 Blackwell (96 GB, compute capability 12.0 / "sm120"), or another Blackwell card (RTX 50-series, B200).
• Linux + a recent NVIDIA driver (580.x+, CUDA 13 capable).
• ~60 GB free disk and a willingness to read one error message carefully.

Check your card first — this skill's performance claims depend on native FP4 tensor cores, which only exist on Blackwell:

nvidia-smi --query-gpu=name,memory.total,driver_version,compute_cap --format=csv
# NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 97887 MiB, 580.159.03, 12.0

If compute_cap is 8.x or 9.x (Ampere / Ada / Hopper), NVFP4 is emulated and the speed table below does not transfer. Use an FP8 or AWQ checkpoint instead.

Concepts used below (skip if familiar)

1. Pick the checkpoint — check for an MTP head BEFORE downloading 50 GB

Some quantized mirrors silently drop the MTP head, costing you the free decode speedup. Check in one command:

REPO="Jiunsong/SuperQwen3.8-27b-abliterated-NVFP4-DGX-Spark"
curl -sL "https://huggingface.co/$REPO/raw/main/model.safetensors.index.json" \
  | python3 -c "import sys,json;wm=json.load(sys.stdin)['weight_map'];print('mtp tensors:',sum(1 for k in wm if k.startswith('mtp.')))"
# expect: 15   (for Qwen3.8-27B)

0 means speculative decoding is gone. Some mirrors ship MTP as a separate model-mtp-bf16.safetensors (kept at BF16 on purpose — quantizing a draft head hurts acceptance) — that is also fine, just confirm the file exists.

Engine rule: model fits in VRAM → vLLM. Model far exceeds VRAM and is MoE → an expert-offload engine instead. Don't use an offload engine on a model that fits; the offload machinery buys nothing and some drop the MTP head.

Download and verify integrity

pip install -U "huggingface_hub[cli]"
hf download "$REPO" --local-dir ~/models/qwen38-nvfp4

If the repo ships SHA256SUMS.json, verify it — a truncated shard produces a confusing CUDA error hours later:

cd ~/models/qwen38-nvfp4 && python3 - <<'PY'
import json, hashlib, os
sums = json.load(open("SHA256SUMS.json"))
cand = sums.get("files") or sums
ok = bad = 0
for name, exp in (cand.items() if isinstance(cand, dict) else []):
    if isinstance(exp, dict): exp = exp.get("sha256", "")
    if not name.endswith(".safetensors") or not os.path.exists(name): continue
    h = hashlib.sha256()
    with open(name, "rb") as f:
        for b in iter(lambda: f.read(1 << 22), b""): h.update(b)
    good = h.hexdigest() == exp
    ok, bad = ok + good, bad + (not good)
    print(("OK   " if good else "BAD  ") + name)
print(f"RESULT ok={ok} bad={bad}")
PY

2. Install vLLM

python3 -m venv ~/vllm-venv
~/vllm-venv/bin/pip install --upgrade pip
~/vllm-venv/bin/pip install vllm==0.27.1     # pulls torch + full CUDA 13 wheels

Do not rely on the system CUDA toolkit. Distro nvidia-cuda-toolkit packages are frequently 12.x and do not know sm_120. That single fact causes trap #1.

3. The three traps (all three will bite on Blackwell)

Trap 1 — FlashInfer requires GPUs with sm75 or higher on an sm120 card

The message is a red herring. FlashInfer JIT-compiles kernels and derives its target architecture list from whichever nvcc is on PATH. A system nvcc 12.x doesn't know sm_120, yields an empty arch list, and the generic "sm75" error is printed. Your GPU is fine.

CUDA_ROOT=$(dirname "$(dirname "$(find ~/vllm-venv/lib/python*/site-packages/nvidia -name nvcc -type f | head -1)")")
export CUDA_HOME="$CUDA_ROOT"
export PATH="$HOME/vllm-venv/bin:$CUDA_ROOT/bin:$PATH"
export TORCH_CUDA_ARCH_LIST=12.0     # 12.0 = Blackwell

~/vllm-venv/bin must be on PATH too — FlashInfer shells out to ninja, which pip installed there. Running $VENV/bin/vllm directly does not activate the venv.

Trap 2 — CUDA compiler and CUDA toolkit headers are incompatible

nvidia-cuda-nvcc and nvidia-cuda-runtime resolved to different minor versions (e.g. 13.3 vs 13.0). CCCL hard-refuses the mismatch.

~/vllm-venv/bin/nvcc --version                        # note the minor, e.g. 13.3
~/vllm-venv/bin/pip install 'nvidia-cuda-runtime==13.3.29'
rm -rf ~/.cache/flashinfer                            # MUST clear — failed objects are cached

Use nvidia-cuda-runtime / nvidia-cuda-nvcc (not the -cu13-suffixed variants, which are stub 0.0.1 packages).

Trap 3 — the sampler calls FlashInfer independently

Even with a different attention backend selected, topk_topp_sampler.py calls FlashInfer on its own path:

export VLLM_USE_FLASHINFER_SAMPLER=0

4. The serve script

Save as ~/qwen38_serve.sh, chmod +x. Templates ship in scripts/.

#!/usr/bin/env bash
set -euo pipefail
MODEL=${MODEL:-$HOME/models/qwen38-nvfp4}
VENV=${VENV:-$HOME/vllm-venv}
CUDA_ROOT=$VENV/lib/python3.12/site-packages/nvidia/cu13
PORT=${PORT:-8000}
CTX=${CTX:-262144}
SEQS=${SEQS:-3}
KVDTYPE=${KVDTYPE:-fp8}
UTIL=${UTIL:-0.90}
YARN=${YARN:-0}
NATIVE=262144            # model's TRAINED window — see config.json

export VLLM_USE_V1=1
export CUDA_VISIBLE_DEVICES=0
export CUDA_HOME=$CUDA_ROOT
export PATH=$VENV/bin:$CUDA_ROOT/bin:$PATH
export TORCH_CUDA_ARCH_LIST=${TORCH_CUDA_ARCH_LIST:-12.0}
export VLLM_USE_FLASHINFER_SAMPLER=${VLLM_USE_FLASHINFER_SAMPLER:-0}

EXTRA=()
if [ "$YARN" = "1" ]; then
  FACTOR=$(python3 -c "print(round($CTX/$NATIVE, 4))")
  EXTRA+=(--hf-overrides "{\"text_config\":{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":$FACTOR,\"original_max_position_embeddings\":$NATIVE}}}")
  echo "[serve] YaRN ENABLED factor=$FACTOR target_ctx=$CTX (native $NATIVE)" >&2
fi

exec "$VENV/bin/vllm" serve "$MODEL" \
  --served-model-name qwen3.8-27b \
  --host 0.0.0.0 --port "$PORT" \
  --max-model-len "$CTX" \
  --max-num-seqs "$SEQS" \
  --gpu-memory-utilization "$UTIL" \
  --kv-cache-dtype "$KVDTYPE" \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  "${EXTRA[@]}"

Notes that cost real time to learn:

• --disable-log-requests was removed in vLLM 0.27 — passing it aborts startup.
• First boot JIT-compiles FlashInfer kernels: allow 5-8 minutes before declaring failure. Later boots are ~90 s.
• --tool-call-parser hermes is WRONG for Qwen3.8 and fails silently — see §6.

5. Run it as a service (survives reboot)

# ~/.config/systemd/user/qwen38-serve.service
[Unit]
Description=Qwen3.8-27B NVFP4 vLLM server
After=network.target

[Service]
Type=simple
ExecStart=%h/qwen38_serve.sh
Environment="CTX=262144"
Environment="UTIL=0.90"
Environment="SEQS=3"
Environment="KVDTYPE=fp8"
Environment="YARN=0"
Restart=on-failure
RestartSec=20
TimeoutStartSec=900
[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now qwen38-serve.service
sudo loginctl enable-linger "$USER"    # ESSENTIAL — see below

enable-linger is not optional. Without it, systemd --user units start on login and are killed on logout — your "autostart" server silently never comes back after a reboot. Verify: loginctl show-user "$USER" --property=Linger → Linger=yes.

Two more service traps that produce genuinely confusing symptoms:

1. systemd --user keeps the OLD Environment= until daemon-reload. After editing a unit, systemctl --user start silently runs with stale env. Always reload first, then verify against the live process, not the unit file:

   tr '\0' ' ' < /proc/$(systemctl --user show qwen38-serve -p MainPID --value)/cmdline
   

2. Audit for OTHER enabled units binding the same port. A leftover unit with Restart=on-failure pointing at a deleted model directory will retry-loop forever and fight for the port on the next reboot — which is precisely when you're not watching. Check what actually starts at boot:

   ls ~/.config/systemd/user/default.target.wants/
   systemctl --user disable <stale-unit>.service
   

Health check

until [ "$(curl -s -o /dev/null -w '%{http_code}' localhost:8000/health)" = 200 ]; do sleep 10; done
curl -s localhost:8000/v1/models | python3 -m json.tool

Trust vLLM's own startup lines over your own arithmetic:

Available KV cache memory: 63.24 GiB
GPU KV cache size: 1,845,614 tokens
Maximum concurrency for 262,144 tokens per request: 7.04x

6. Tool calling — verify it, never assume it

--tool-call-parser hermes silently fails on Qwen3.8. Qwen3.8 emits XML-style calls; the hermes parser expects JSON inside <tool_call>. The failure mode is nasty: HTTP 200, finish_reason: "stop", tool_calls: null, and the raw XML dumped into content. Your agent sees prose where a tool call should be and the loop dies with no error anywhere.

Use --tool-call-parser qwen3_coder, then prove it with a real call:

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"qwen3.8-27b",
  "messages":[{"role":"user","content":"Weather in Krakow? Use the tool."}],
  "tools":[{"type":"function","function":{"name":"get_weather",
    "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
  "tool_choice":"auto"}' | python3 -m json.tool

PASS = finish_reason: "tool_calls" and a populated tool_calls array. Anything else is a fail, even with HTTP 200.

File name ≠ registered name. vLLM 0.27.1 registers the tool parsers qwen3_coder, qwen3_xml, qwen3_engine; for reasoning the only registered name is qwen3 (--reasoning-parser qwen3_engine is a KeyError that kills startup).

7. Controlling "thinking" (reasoning effort)

Qwen3.8's ladder is low | medium | xhigh. Sending OpenAI's "high" returns HTTP 400.

// per request
{"chat_template_kwargs": {"reasoning_effort": "xhigh"}}
{"chat_template_kwargs": {"enable_thinking": false}}   // off entirely

# server-wide default
--default-chat-template-kwargs '{"reasoning_effort":"medium"}'

Find out what your default ACTUALLY is (don't guess)

The effort level is implemented as a system preamble injected by the chat template. Render the prompt and read it — deterministic, no sampling noise:

curl -s localhost:8000/tokenize -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"hi"}],"return_token_strs":true}'

⚠️ /tokenize is at the ROOT, not under /v1. POST /v1/tokenize returns 404 {"detail":"Not Found"}, which reads exactly like a broken server.

Compare token counts across settings — identical counts mean identical prompts:

Measured effect (tok/s is nearly flat — effort changes how many tokens are generated, not how fast):

Default to medium.

⚠️ xhigh runaway is REAL and REPRODUCIBLE — measured, not theoretical

On a hard prompt (Manacher's algorithm), reasoning_effort=xhigh regularly consumes the entire token budget and returns zero answer characters — the model thinks until it hits the cap and never answers. Raising the budget does not reliably fix it:

The failure signature is finish_reason: "length" + completion_tokens == max_tokens + content: "". Note this checkpoint's own card advertises a "36/36 overthinking pass" — that gate did not survive contact with this prompt on this hardware. Measure it yourself; don't inherit a vendor's claim.

Practical rules:

1. Leave the server default at medium. It was stable 3/3 and answers just as correctly.
2. Treat xhigh as opt-in for short, well-bounded prompts only.
3. Any client using xhigh must handle the empty-answer case — check for finish_reason == "length" with empty content and retry at medium.
4. Benchmark honestly: if finish_reason == "length", the run was capped and every effort level looks identical. Always confirm stop.

Because runaway is intermittent, a single passing trial proves nothing — run at least 3.

Empty reasoning_content? Diagnose the layer before swapping models

reasoning_content: "" while hundreds of tokens were billed has three possible causes, and two of them are not the model's fault. Three calls settle it:

# 1. is </think> a single token the parser can match?
curl -s localhost:8000/tokenize -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b","prompt":"</think>","return_token_strs":true,"add_special_tokens":false}'
#    => {"count":1,...} good.

# 2. render the REAL prompt to token IDs, feed those IDs to raw /v1/completions
ids = post("/tokenize", {"model": M, "messages": msgs,
                         "chat_template_kwargs": {"enable_thinking": True}})["tokens"]
raw = post("/v1/completions", {"model": M, "prompt": ids, "max_tokens": 6000})
"</think>" in raw["choices"][0]["text"]      # True => the WEIGHTS are fine

# 3. same question via /v1/chat/completions
len(chat["choices"][0]["message"].get("reasoning_content") or "")   # 0 => PARSER

Never hand-build a chat prompt as a Python string for /v1/completions. Qwen's tokenizer is byte-level BPE (newline = Ċ); a hand-typed "<|im_start|>user\n…<think>\n" can tokenize to ~1 token and return an empty generation — which looks exactly like "the model refuses to think." Always render via /tokenize and pass the returned token id list as prompt.

Known result on vLLM 0.27.1 + Qwen3.8: raw emits 5,393 chars (medium) / 11,741 chars (xhigh) of reasoning, while chat returns reasoning_content = 0 for every combination of enable_thinking / reasoning_effort / streaming. Impact is cosmetic — the thinking still happens and still improves the answer, it just lands in content instead of a separate field. A chars_per_token < 1.5 ratio in the chat response is the cheap smell test.

8. Vision

The checkpoint is image-text-to-text; the vision tower survives NVFP4 (the quantizer excludes visual, lm_head, conv1d, mtp).

A 422 Failed to fetch media from URL is not a dead vision tower — it means the server couldn't fetch your URL (egress rules, CDN rejection). Re-test with an inline base64 data URL before concluding anything:

python3 scripts/probe_vision.py           # generates a solid-colour PNG, asks the colour

9. KV-cache math (hybrid Gated-DeltaNet models)

Qwen3.5/3.6/3.8 are hybrid: only full-attention layers hold a growing KV cache; linear-attention (GDN) layers hold a fixed-size recurrent state.

L_full     = num_hidden_layers / full_attention_interval     # 64 / 4 = 16
kv_per_tok = L_full * num_key_value_heads * head_dim * 2 * bytes_per_elem

Qwen3.8-27B: 16 × 4 × 256 × 2 × 2B = 64 KiB/token at bf16 → 17.2 GB per 256k sequence. --kv-cache-dtype fp8 halves it. Always trust vLLM's printed Maximum concurrency line over your arithmetic.

Why NVFP4 is a double win on Blackwell

Blackwell has native FP4 tensor cores — that's the 2×. On Ampere/Ada it would be emulated and this table does not transfer.

10. Optional: a 1M-token context window with YaRN

max_position_embeddings is the trained window, not a hard ceiling. vLLM can stretch it at load time. For hybrid Qwen3.5/3.8 the scaling must go under text_config — placing rope_scaling at the JSON root is silently ignored:

CTX=1048576 YARN=1 SEQS=3 UTIL=0.92 ~/qwen38_serve.sh

Measured (RTX PRO 6000 96 GB, NVFP4, fp8 KV, util=0.92):

⚠️ Prefill is super-linear: 3.94× the tokens cost 12.4× the wall time. Budget ~15 minutes before the first token at 1M. Decode afterwards is unaffected (98.9 tok/s @1M vs 101.3 @262k — noise). Treat 1M as "load a whole codebase once and interrogate it", not an everyday window.

max_num_seqs does not have to drop. vLLM allocates KV blocks by tokens actually used, not by max_model_len, so a 1M ceiling with SEQS=3 costs nothing on ordinary short traffic.

Always run a SHORT-context regression after enabling YaRN

RoPE scaling can tax short-prompt quality. Test both sides or you ship a 1M window that got worse at the 4k work you actually do:

python3 scripts/short_context_regression.py

Reference result: 5/6 thinking-off, 5/5 thinking-on, tools PASS, 98.9 tok/s. The single miss was 84*37-1000 answered with thinking disabled — the same class of mental-arithmetic slip a non-YaRN model makes, self-corrected with thinking on. Don't score a no-think arithmetic slip as a YaRN regression; re-run the failing item with enable_thinking: true before blaming rope scaling.

Rollback is a drop-in file, not an edit

Keep extended-context settings in a systemd drop-in so the base unit stays pristine:

# ~/.config/systemd/user/qwen38-serve.service.d/yarn1m.conf
[Service]
Environment="CTX=1048576"
Environment="YARN=1"
Environment="SEQS=3"
Environment="UTIL=0.92"

Deleting that one file + daemon-reload + restart returns you to native. No edits to undo.

11. Prove the window is USABLE, not merely accepted

A server that accepts 1M tokens has proven nothing. Retrieval is the test:

python3 scripts/needle_test.py 262000      # native
python3 scripts/needle_test.py 1040000     # YaRN

The script buries a distinctive sentence mid-haystack, trims the prompt to the target token count using the server's own tokenizer, and checks the answer verbatim. Anything less is theatre.

12. Connect any harness

The result is a plain OpenAI-compatible endpoint. Nothing here is framework-specific.

export OPENAI_BASE_URL="http://<host>:8000/v1"
export OPENAI_API_KEY="not-needed"      # local vLLM has no auth by default

from openai import OpenAI
client = OpenAI(base_url="http://<host>:8000/v1", api_key="not-needed")
print(client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[{"role": "user", "content": "Reply with exactly: OK"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
).choices[0].message.content)

Client notes:

• Reasoning-parsed models can return content: null with text in reasoning_content. Handle both fields, or send chat_template_kwargs: {"enable_thinking": false} for direct answers.
• Bind --host 0.0.0.0 only on a trusted network. There is no auth. For remote access use a VPN/overlay network, an SSH tunnel, or put a reverse proxy with auth in front. Do not expose port 8000 to the internet.

Verify at three levels — do all three

1. Config/endpoint: /health returns 200 and /v1/models lists your name.
2. Real completion: a POST /v1/chat/completions returns sane text.
3. Your actual harness runs a turn end-to-end. Levels 1-2 pass on setups your framework still cannot use (wrong model alias, wrong context length in client config). Only level 3 proves the wiring.

Pitfalls

• nvidia-smi right after a restart shows the OLD process's memory while it tears down. Wait for /health, then re-read, or you'll "fix" a non-problem.
• An orphaned EngineCore can hold the whole GPU after its parent dies — systemctl stop and pkill -f 'vllm serve' both miss it, so the next boot fails on free memory. Find the real holder: nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader.
• Free memory on device … is less than desired GPU memory utilization means another process holds VRAM — not a vLLM bug.
• A model killed with kill <pid> that comes straight back is being restarted by a systemd unit. systemctl --user list-units | grep -i vllm, then disable.
• Write systemd units locally and scp them. Heredocs piped through ssh "..." get mangled and silently produce a partial unit.
• Long benchmarks over ssh die with client timeouts. Use setsid nohup python3 -u script.py > log 2>&1 < /dev/null & (note -u, or Python buffers and the log stays empty), then poll the log.
• Don't trust a social-media context claim over the repo. One checkpoint was announced as "1M context" while its own model card said the 1M run "was explicitly stopped before completion and is not claimed as a pass", shipped a waiver file, set rope_scaling: null, and declared max_position_embeddings: 262144. (You can still reach 1M with §10 — but that's your YaRN config working, not a shipped model capability.) Resolve claims against config.json first.

License

MIT. Verified on RTX PRO 6000 Blackwell (96 GB, sm120), driver 580.159.03, vLLM 0.27.1, 2026-08-25.