# Qwen38 Vllm Rtx Pro 6000

> Serving Qwen3.8-27B on an RTX PRO 6000 Blackwell with vLLM

- Skill: `wisechef-ai/qwen38-vllm-rtx-pro-6000` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add wisechef-ai/qwen38-vllm-rtx-pro-6000`
- Raw SKILL.md: https://api.skillmd.com/api/skills/wisechef-ai/qwen38-vllm-rtx-pro-6000/raw
- Safety review: pending (external: skill-scanner WARNING, skillspector FAIL)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: wisechef-ai (https://skillmd.com/u/wisechef-ai)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/wisechef-ai/qwen38-vllm-rtx-pro-6000

---


# Serving Qwen3.8-27B on an RTX PRO 6000 Blackwell with vLLM

Stand up a production OpenAI-compatible endpoint on one Blackwell card: NVFP4
weights, built-in speculative decoding, working tool calls, vision, and an
optional **1M-token** context window.

Every number in this skill was measured on real hardware, not estimated.
Everything here is plain vLLM + systemd — **no agent framework required**. Any
client that speaks the OpenAI API (LangChain, LlamaIndex, Open WebUI, Cline,
Continue, a raw `curl`) works against the result.

---

## 0. Who this is for

You have:

- An **NVIDIA RTX PRO 6000 Blackwell** (96 GB, compute capability **12.0** /
  "sm120"), or another Blackwell card (RTX 50-series, B200).
- Linux + a recent NVIDIA driver (**580.x+**, CUDA 13 capable).
- ~60 GB free disk and a willingness to read one error message carefully.

Check your card first — this skill's performance claims depend on **native FP4
tensor cores**, which only exist on Blackwell:

```bash
nvidia-smi --query-gpu=name,memory.total,driver_version,compute_cap --format=csv
# NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 97887 MiB, 580.159.03, 12.0
```

If `compute_cap` is **8.x or 9.x** (Ampere / Ada / Hopper), NVFP4 is *emulated*
and the speed table below does not transfer. Use an FP8 or AWQ checkpoint instead.

### Concepts used below (skip if familiar)

| Term | Plain meaning |
|---|---|
| **NVFP4** | 4-bit float weights **and** activations. On Blackwell the silicon runs these natively — smaller *and* faster, not a tradeoff. |
| **MTP** | Multi-Token Prediction. A small extra head that guesses the next few tokens so the big model can verify several at once — free speculative decoding, ~1.5-2.5× decode. |
| **KV cache** | Per-token memory holding attention state. Determines how much context you can hold and how many requests run concurrently. |
| **YaRN** | A RoPE-scaling trick that stretches a model's context window *beyond* what it was trained on. |
| **Prefill vs decode** | Prefill = reading your prompt (one big parallel pass). Decode = writing the answer (one token at a time). They scale very differently — this matters a lot at 1M context. |

---

## 1. Pick the checkpoint — check for an MTP head BEFORE downloading 50 GB

Some quantized mirrors silently drop the MTP head, costing you the free decode
speedup. Check in one command:

```bash
REPO="Jiunsong/SuperQwen3.8-27b-abliterated-NVFP4-DGX-Spark"
curl -sL "https://huggingface.co/$REPO/raw/main/model.safetensors.index.json" \
  | python3 -c "import sys,json;wm=json.load(sys.stdin)['weight_map'];print('mtp tensors:',sum(1 for k in wm if k.startswith('mtp.')))"
# expect: 15   (for Qwen3.8-27B)
```

`0` means speculative decoding is gone. Some mirrors ship MTP as a separate
`model-mtp-bf16.safetensors` (kept at BF16 on purpose — quantizing a draft head
hurts acceptance) — that is also fine, just confirm the file exists.

**Engine rule:** model *fits* in VRAM → **vLLM**. Model far exceeds VRAM and is
MoE → an expert-offload engine instead. Don't use an offload engine on a model
that fits; the offload machinery buys nothing and some drop the MTP head.

### Download and verify integrity

```bash
pip install -U "huggingface_hub[cli]"
hf download "$REPO" --local-dir ~/models/qwen38-nvfp4
```

If the repo ships `SHA256SUMS.json`, **verify it** — a truncated shard produces a
confusing CUDA error hours later:

```bash
cd ~/models/qwen38-nvfp4 && python3 - <<'PY'
import json, hashlib, os
sums = json.load(open("SHA256SUMS.json"))
cand = sums.get("files") or sums
ok = bad = 0
for name, exp in (cand.items() if isinstance(cand, dict) else []):
    if isinstance(exp, dict): exp = exp.get("sha256", "")
    if not name.endswith(".safetensors") or not os.path.exists(name): continue
    h = hashlib.sha256()
    with open(name, "rb") as f:
        for b in iter(lambda: f.read(1 << 22), b""): h.update(b)
    good = h.hexdigest() == exp
    ok, bad = ok + good, bad + (not good)
    print(("OK   " if good else "BAD  ") + name)
print(f"RESULT ok={ok} bad={bad}")
PY
```

---

## 2. Install vLLM

```bash
python3 -m venv ~/vllm-venv
~/vllm-venv/bin/pip install --upgrade pip
~/vllm-venv/bin/pip install vllm==0.27.1     # pulls torch + full CUDA 13 wheels
```

**Do not rely on the system CUDA toolkit.** Distro `nvidia-cuda-toolkit` packages
are frequently 12.x and do not know `sm_120`. That single fact causes trap #1.

---

## 3. The three traps (all three will bite on Blackwell)

### Trap 1 — `FlashInfer requires GPUs with sm75 or higher` on an sm120 card

The message is a **red herring**. FlashInfer JIT-compiles kernels and derives its
target architecture list from **whichever `nvcc` is on `PATH`**. A system nvcc
12.x doesn't know `sm_120`, yields an empty arch list, and the generic "sm75"
error is printed. Your GPU is fine.

```bash
CUDA_ROOT=$(dirname "$(dirname "$(find ~/vllm-venv/lib/python*/site-packages/nvidia -name nvcc -type f | head -1)")")
export CUDA_HOME="$CUDA_ROOT"
export PATH="$HOME/vllm-venv/bin:$CUDA_ROOT/bin:$PATH"
export TORCH_CUDA_ARCH_LIST=12.0     # 12.0 = Blackwell
```

`~/vllm-venv/bin` must be on `PATH` too — FlashInfer shells out to `ninja`, which
pip installed there. Running `$VENV/bin/vllm` directly does **not** activate the venv.

### Trap 2 — `CUDA compiler and CUDA toolkit headers are incompatible`

`nvidia-cuda-nvcc` and `nvidia-cuda-runtime` resolved to different minor versions
(e.g. 13.3 vs 13.0). CCCL hard-refuses the mismatch.

```bash
~/vllm-venv/bin/nvcc --version                        # note the minor, e.g. 13.3
~/vllm-venv/bin/pip install 'nvidia-cuda-runtime==13.3.29'
rm -rf ~/.cache/flashinfer                            # MUST clear — failed objects are cached
```

Use `nvidia-cuda-runtime` / `nvidia-cuda-nvcc` (**not** the `-cu13`-suffixed
variants, which are stub 0.0.1 packages).

### Trap 3 — the sampler calls FlashInfer independently

Even with a different attention backend selected, `topk_topp_sampler.py` calls
FlashInfer on its own path:

```bash
export VLLM_USE_FLASHINFER_SAMPLER=0
```

---

## 4. The serve script

Save as `~/qwen38_serve.sh`, `chmod +x`. Templates ship in `scripts/`.

```bash
#!/usr/bin/env bash
set -euo pipefail
MODEL=${MODEL:-$HOME/models/qwen38-nvfp4}
VENV=${VENV:-$HOME/vllm-venv}
CUDA_ROOT=$VENV/lib/python3.12/site-packages/nvidia/cu13
PORT=${PORT:-8000}
CTX=${CTX:-262144}
SEQS=${SEQS:-3}
KVDTYPE=${KVDTYPE:-fp8}
UTIL=${UTIL:-0.90}
YARN=${YARN:-0}
NATIVE=262144            # model's TRAINED window — see config.json

export VLLM_USE_V1=1
export CUDA_VISIBLE_DEVICES=0
export CUDA_HOME=$CUDA_ROOT
export PATH=$VENV/bin:$CUDA_ROOT/bin:$PATH
export TORCH_CUDA_ARCH_LIST=${TORCH_CUDA_ARCH_LIST:-12.0}
export VLLM_USE_FLASHINFER_SAMPLER=${VLLM_USE_FLASHINFER_SAMPLER:-0}

EXTRA=()
if [ "$YARN" = "1" ]; then
  FACTOR=$(python3 -c "print(round($CTX/$NATIVE, 4))")
  EXTRA+=(--hf-overrides "{\"text_config\":{\"rope_scaling\":{\"rope_type\":\"yarn\",\"factor\":$FACTOR,\"original_max_position_embeddings\":$NATIVE}}}")
  echo "[serve] YaRN ENABLED factor=$FACTOR target_ctx=$CTX (native $NATIVE)" >&2
fi

exec "$VENV/bin/vllm" serve "$MODEL" \
  --served-model-name qwen3.8-27b \
  --host 0.0.0.0 --port "$PORT" \
  --max-model-len "$CTX" \
  --max-num-seqs "$SEQS" \
  --gpu-memory-utilization "$UTIL" \
  --kv-cache-dtype "$KVDTYPE" \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  "${EXTRA[@]}"
```

Notes that cost real time to learn:

- `--disable-log-requests` was **removed** in vLLM 0.27 — passing it aborts startup.
- First boot JIT-compiles FlashInfer kernels: allow **5-8 minutes** before
  declaring failure. Later boots are ~90 s.
- **`--tool-call-parser hermes` is WRONG for Qwen3.8** and fails *silently* — see §6.

---

## 5. Run it as a service (survives reboot)

```ini
# ~/.config/systemd/user/qwen38-serve.service
[Unit]
Description=Qwen3.8-27B NVFP4 vLLM server
After=network.target

[Service]
Type=simple
ExecStart=%h/qwen38_serve.sh
Environment="CTX=262144"
Environment="UTIL=0.90"
Environment="SEQS=3"
Environment="KVDTYPE=fp8"
Environment="YARN=0"
Restart=on-failure
RestartSec=20
TimeoutStartSec=900
[Install]
WantedBy=default.target
```

```bash
systemctl --user daemon-reload
systemctl --user enable --now qwen38-serve.service
sudo loginctl enable-linger "$USER"    # ESSENTIAL — see below
```

**`enable-linger` is not optional.** Without it, `systemd --user` units start on
*login* and are killed on logout — your "autostart" server silently never comes
back after a reboot. Verify: `loginctl show-user "$USER" --property=Linger` → `Linger=yes`.

Two more service traps that produce genuinely confusing symptoms:

1. **`systemd --user` keeps the OLD `Environment=` until `daemon-reload`.** After
   editing a unit, `systemctl --user start` silently runs with stale env. Always
   reload first, then verify against the **live process**, not the unit file:
   ```bash
   tr '\0' ' ' < /proc/$(systemctl --user show qwen38-serve -p MainPID --value)/cmdline
   ```
2. **Audit for OTHER enabled units binding the same port.** A leftover unit with
   `Restart=on-failure` pointing at a deleted model directory will retry-loop
   forever and fight for the port *on the next reboot* — which is precisely when
   you're not watching. Check what actually starts at boot:
   ```bash
   ls ~/.config/systemd/user/default.target.wants/
   systemctl --user disable <stale-unit>.service
   ```

### Health check

```bash
until [ "$(curl -s -o /dev/null -w '%{http_code}' localhost:8000/health)" = 200 ]; do sleep 10; done
curl -s localhost:8000/v1/models | python3 -m json.tool
```

Trust vLLM's own startup lines over your own arithmetic:

```
Available KV cache memory: 63.24 GiB
GPU KV cache size: 1,845,614 tokens
Maximum concurrency for 262,144 tokens per request: 7.04x
```

---

## 6. Tool calling — verify it, never assume it

**`--tool-call-parser hermes` silently fails on Qwen3.8.** Qwen3.8 emits
XML-style calls; the `hermes` parser expects JSON inside `<tool_call>`. The
failure mode is nasty: HTTP 200, `finish_reason: "stop"`, `tool_calls: null`,
and the raw XML dumped into `content`. Your agent sees prose where a tool call
should be and the loop dies with no error anywhere.

**Use `--tool-call-parser qwen3_coder`,** then prove it with a real call:

```bash
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"qwen3.8-27b",
  "messages":[{"role":"user","content":"Weather in Krakow? Use the tool."}],
  "tools":[{"type":"function","function":{"name":"get_weather",
    "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
  "tool_choice":"auto"}' | python3 -m json.tool
```

**PASS** = `finish_reason: "tool_calls"` **and** a populated `tool_calls` array.
Anything else is a fail, even with HTTP 200.

> File name ≠ registered name. vLLM 0.27.1 registers the tool parsers
> `qwen3_coder`, `qwen3_xml`, `qwen3_engine`; for **reasoning** the only
> registered name is `qwen3` (`--reasoning-parser qwen3_engine` is a `KeyError`
> that kills startup).

---

## 7. Controlling "thinking" (reasoning effort)

Qwen3.8's ladder is **`low` | `medium` | `xhigh`**. Sending OpenAI's `"high"`
returns **HTTP 400**.

```jsonc
// per request
{"chat_template_kwargs": {"reasoning_effort": "xhigh"}}
{"chat_template_kwargs": {"enable_thinking": false}}   // off entirely
```

```bash
# server-wide default
--default-chat-template-kwargs '{"reasoning_effort":"medium"}'
```

### Find out what your default ACTUALLY is (don't guess)

The effort level is implemented as a **system preamble injected by the chat
template**. Render the prompt and read it — deterministic, no sampling noise:

```bash
curl -s localhost:8000/tokenize -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"hi"}],"return_token_strs":true}'
```

⚠️ **`/tokenize` is at the ROOT, not under `/v1`.** `POST /v1/tokenize` returns
`404 {"detail":"Not Found"}`, which reads exactly like a broken server.

Compare token counts across settings — identical counts mean identical prompts:

| sent | preamble | prompt tokens |
|---|---|---|
| nothing | none | 11 |
| `medium` | none | **11** ← identical to default |
| `low` | "Reasoning effort is set to low…" | 59 |
| `xhigh` | "Reasoning effort is set to xhigh…" | 90 |
| `enable_thinking:false` | none, prefills `<think>\n\n</think>` | 13 |

Measured effect (tok/s is nearly flat — effort changes **how many** tokens are
generated, not how fast):

| effort | tok/s | gen tokens | wall |
|---|---|---|---|
| low | 65.4 | 1610 | 55% |
| medium | 65.7 | 1982 | 68% |
| xhigh | 62.4 | 2679 | 100% |

**Default to `medium`.**

### ⚠️ `xhigh` runaway is REAL and REPRODUCIBLE — measured, not theoretical

On a hard prompt (Manacher's algorithm), `reasoning_effort=xhigh` regularly
consumes the entire token budget and returns **zero answer characters** — the
model thinks until it hits the cap and never answers. Raising the budget does
**not** reliably fix it:

| effort | budget | answered | runaway |
|---|---|---|---|
| **medium** | 8,000 | **3/3** | 0/3 |
| xhigh | 8,000 | 1/3 | **2/3** |
| xhigh | 16,000 | 2/3 | **1/3** |

The failure signature is `finish_reason: "length"` + `completion_tokens ==
max_tokens` + `content: ""`. Note this checkpoint's own card advertises a
"36/36 overthinking pass" — that gate did not survive contact with this prompt
on this hardware. **Measure it yourself; don't inherit a vendor's claim.**

Practical rules:

1. **Leave the server default at `medium`.** It was stable 3/3 and answers just
   as correctly.
2. Treat `xhigh` as opt-in for short, well-bounded prompts only.
3. **Any client using `xhigh` must handle the empty-answer case** — check for
   `finish_reason == "length"` with empty `content` and retry at `medium`.
4. Benchmark honestly: if `finish_reason == "length"`, the run was *capped* and
   every effort level looks identical. Always confirm `stop`.

Because runaway is intermittent, a **single passing trial proves nothing** —
run at least 3.


### Empty `reasoning_content`? Diagnose the layer before swapping models

`reasoning_content: ""` while hundreds of tokens were billed has **three**
possible causes, and two of them are not the model's fault. Three calls settle it:

```bash
# 1. is </think> a single token the parser can match?
curl -s localhost:8000/tokenize -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b","prompt":"</think>","return_token_strs":true,"add_special_tokens":false}'
#    => {"count":1,...} good.
```

```python
# 2. render the REAL prompt to token IDs, feed those IDs to raw /v1/completions
ids = post("/tokenize", {"model": M, "messages": msgs,
                         "chat_template_kwargs": {"enable_thinking": True}})["tokens"]
raw = post("/v1/completions", {"model": M, "prompt": ids, "max_tokens": 6000})
"</think>" in raw["choices"][0]["text"]      # True => the WEIGHTS are fine
```

```python
# 3. same question via /v1/chat/completions
len(chat["choices"][0]["message"].get("reasoning_content") or "")   # 0 => PARSER
```

| raw has `</think>` | chat `reasoning_content` | verdict |
|---|---|---|
| ✅ | > 0 | healthy |
| ✅ | **0** | **vLLM parser bug** — swapping checkpoints will NOT fix it |
| ❌ | 0 | the checkpoint's ablation broke the stop token |

> **Never hand-build a chat prompt as a Python string for `/v1/completions`.**
> Qwen's tokenizer is byte-level BPE (newline = `Ċ`); a hand-typed
> `"<|im_start|>user\n…<think>\n"` can tokenize to ~1 token and return an empty
> generation — which looks exactly like "the model refuses to think." Always
> render via `/tokenize` and pass the returned **token id list** as `prompt`.

Known result on vLLM **0.27.1** + Qwen3.8: raw emits 5,393 chars (medium) /
11,741 chars (xhigh) of reasoning, while chat returns `reasoning_content = 0`
for **every** combination of `enable_thinking` / `reasoning_effort` / streaming.
**Impact is cosmetic** — the thinking still happens and still improves the
answer, it just lands in `content` instead of a separate field. A
`chars_per_token < 1.5` ratio in the chat response is the cheap smell test.

---

## 8. Vision

The checkpoint is `image-text-to-text`; the vision tower survives NVFP4 (the
quantizer excludes `visual`, `lm_head`, `conv1d`, `mtp`).

**A `422 Failed to fetch media from URL` is not a dead vision tower** — it means
*the server* couldn't fetch your URL (egress rules, CDN rejection). Re-test with
an inline base64 data URL before concluding anything:

```bash
python3 scripts/probe_vision.py           # generates a solid-colour PNG, asks the colour
```

---

## 9. KV-cache math (hybrid Gated-DeltaNet models)

Qwen3.5/3.6/3.8 are **hybrid**: only *full-attention* layers hold a growing KV
cache; linear-attention (GDN) layers hold a fixed-size recurrent state.

```
L_full     = num_hidden_layers / full_attention_interval     # 64 / 4 = 16
kv_per_tok = L_full * num_key_value_heads * head_dim * 2 * bytes_per_elem
```

Qwen3.8-27B: `16 × 4 × 256 × 2 × 2B` = **64 KiB/token** at bf16 → 17.2 GB per
256k sequence. `--kv-cache-dtype fp8` halves it. Always trust vLLM's printed
`Maximum concurrency` line over your arithmetic.

### Why NVFP4 is a double win on Blackwell

| | BF16 | NVFP4 | delta |
|---|---|---|---|
| weights | 55.6 GB | 20.6 GB | −35 GB |
| decode | 66 tok/s | **132 tok/s** | **2.0×** |
| KV pool | 1.05M tok | 1.53M tok | +46% |
| correctness battery | — | 6/6 | no regression |
| tool calling | ✅ | ✅ | preserved |

Blackwell has native FP4 tensor cores — that's the 2×. On Ampere/Ada it would be
emulated and this table does not transfer.

---

## 10. Optional: a 1M-token context window with YaRN

`max_position_embeddings` is the **trained** window, not a hard ceiling. vLLM can
stretch it at load time. For hybrid Qwen3.5/3.8 the scaling must go under
**`text_config`** — placing `rope_scaling` at the JSON root is **silently ignored**:

```bash
CTX=1048576 YARN=1 SEQS=3 UTIL=0.92 ~/qwen38_serve.sh
```

**Measured** (RTX PRO 6000 96 GB, NVFP4, fp8 KV, `util=0.92`):

| ctx | prompt tokens | prefill wall | needle retrieved | concurrency |
|---|---|---|---|---|
| 262,144 native | 260,549 | **74 s** | ✅ exact | 7.04× |
| 1,048,576 YaRN ×4 | **1,026,629** | **925 s** | ✅ exact | 1.88× |

⚠️ **Prefill is super-linear: 3.94× the tokens cost 12.4× the wall time.** Budget
~15 minutes before the first token at 1M. Decode afterwards is unaffected
(98.9 tok/s @1M vs 101.3 @262k — noise). Treat 1M as "load a whole codebase once
and interrogate it", not an everyday window.

**`max_num_seqs` does not have to drop.** vLLM allocates KV blocks by tokens
*actually used*, not by `max_model_len`, so a 1M ceiling with `SEQS=3` costs
nothing on ordinary short traffic.

### Always run a SHORT-context regression after enabling YaRN

RoPE scaling can tax short-prompt quality. Test both sides or you ship a 1M
window that got *worse* at the 4k work you actually do:

```bash
python3 scripts/short_context_regression.py
```

Reference result: **5/6 thinking-off, 5/5 thinking-on**, tools PASS, 98.9 tok/s.
The single miss was `84*37-1000` answered with thinking disabled — the same class
of mental-arithmetic slip a non-YaRN model makes, self-corrected with thinking on.
**Don't score a no-think arithmetic slip as a YaRN regression**; re-run the
failing item with `enable_thinking: true` before blaming rope scaling.

### Rollback is a drop-in file, not an edit

Keep extended-context settings in a systemd drop-in so the base unit stays pristine:

```bash
# ~/.config/systemd/user/qwen38-serve.service.d/yarn1m.conf
[Service]
Environment="CTX=1048576"
Environment="YARN=1"
Environment="SEQS=3"
Environment="UTIL=0.92"
```

Deleting that one file + `daemon-reload` + `restart` returns you to native. No
edits to undo.

---

## 11. Prove the window is USABLE, not merely accepted

A server that *accepts* 1M tokens has proven nothing. Retrieval is the test:

```bash
python3 scripts/needle_test.py 262000      # native
python3 scripts/needle_test.py 1040000     # YaRN
```

The script buries a distinctive sentence mid-haystack, trims the prompt to the
target token count using the server's own tokenizer, and checks the answer
verbatim. Anything less is theatre.

---

## 12. Connect any harness

The result is a plain OpenAI-compatible endpoint. Nothing here is
framework-specific.

```bash
export OPENAI_BASE_URL="http://<host>:8000/v1"
export OPENAI_API_KEY="not-needed"      # local vLLM has no auth by default
```

```python
from openai import OpenAI
client = OpenAI(base_url="http://<host>:8000/v1", api_key="not-needed")
print(client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[{"role": "user", "content": "Reply with exactly: OK"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
).choices[0].message.content)
```

Client notes:

- **Reasoning-parsed models can return `content: null`** with text in
  `reasoning_content`. Handle **both** fields, or send
  `chat_template_kwargs: {"enable_thinking": false}` for direct answers.
- **Bind `--host 0.0.0.0` only on a trusted network.** There is no auth. For
  remote access use a VPN/overlay network, an SSH tunnel, or put a reverse proxy
  with auth in front. Do not expose port 8000 to the internet.

### Verify at three levels — do all three

1. **Config/endpoint:** `/health` returns 200 and `/v1/models` lists your name.
2. **Real completion:** a `POST /v1/chat/completions` returns sane text.
3. **Your actual harness runs a turn end-to-end.** Levels 1-2 pass on setups your
   framework still cannot use (wrong model alias, wrong context length in client
   config). Only level 3 proves the wiring.

---

## Pitfalls

- **`nvidia-smi` right after a restart shows the OLD process's memory** while it
  tears down. Wait for `/health`, then re-read, or you'll "fix" a non-problem.
- **An orphaned `EngineCore` can hold the whole GPU** after its parent dies —
  `systemctl stop` and `pkill -f 'vllm serve'` both miss it, so the next boot
  fails on free memory. Find the real holder:
  `nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader`.
- **`Free memory on device … is less than desired GPU memory utilization`** means
  another process holds VRAM — not a vLLM bug.
- **A model killed with `kill <pid>` that comes straight back** is being restarted
  by a systemd unit. `systemctl --user list-units | grep -i vllm`, then `disable`.
- **Write systemd units locally and `scp` them.** Heredocs piped through
  `ssh "..."` get mangled and silently produce a partial unit.
- **Long benchmarks over `ssh` die with client timeouts.** Use
  `setsid nohup python3 -u script.py > log 2>&1 < /dev/null &` (note `-u`, or
  Python buffers and the log stays empty), then poll the log.
- **Don't trust a social-media context claim over the repo.** One checkpoint was
  announced as "1M context" while its own model card said the 1M run *"was
  explicitly stopped before completion and is not claimed as a pass"*, shipped a
  waiver file, set `rope_scaling: null`, and declared `max_position_embeddings:
  262144`. (You can still *reach* 1M with §10 — but that's your YaRN config
  working, not a shipped model capability.) Resolve claims against `config.json`
  first.

## License

MIT. Verified on RTX PRO 6000 Blackwell (96 GB, sm120), driver 580.159.03,
vLLM 0.27.1, 2026-08-25.