Minicpm5 Deploy Vllm Ascend

Deploy MiniCPM5-2B with vLLM-Ascend

Use this skill for the BF16 / FP16 openbmb/MiniCPM5-2B checkpoint on Huawei Ascend NPU. vllm-ascend is installed alongside a matching vLLM release and supplies the Ascend backend.

Required input

Accept either the Hugging Face id or a local directory containing the MiniCPM5-2B model files.

Prerequisites

Use a compatible Linux Ascend environment with CANN, PyTorch, and TorchNPU installed. Keep the vLLM and vLLM-Ascend versions matched. The basic serving recipe below follows the tested vllm==0.18.0 / vllm-ascend==0.18.0 pair.

Device visibility is controlled by ASCEND_RT_VISIBLE_DEVICES; CUDA_VISIBLE_DEVICES does not select Ascend devices.

Install

The prebuilt image is the simplest option for a compatible Ascend host. This is a minimal single-device example; adjust device nodes and driver mounts for the host hardware and image documentation.

docker run -it --net=host --shm-size=1g \
    --device /dev/davinci0 --device /dev/davinci_manager \
    --device /dev/devmm_svm --device /dev/hisi_hdc \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    quay.io/ascend/vllm-ascend:v0.18.0

For an existing CANN environment:

pip install vllm==0.18.0 vllm-ascend==0.18.0 \
    --extra-index-url https://download.pytorch.org/whl/cpu

For built-in MiniCPM5 tool-call parsing, use the matching vllm==0.23.0 / vllm-ascend==0.23.0 stack described in Tool calling. The parser responses in the cookbook were produced with the upstream parser file on an older vLLM and a compatibility shim because the 0.23.0 stack was not available on the test hardware.

Prepare the environment

Run the CANN and NNAL setup scripts before starting vLLM. Change the paths if those components are installed elsewhere.

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_RT_VISIBLE_DEVICES=${ASCEND_RT_VISIBLE_DEVICES:-0}
export LD_LIBRARY_PATH=$(python3 -c "import torch,os;print(os.path.dirname(torch.__file__))")/lib:\
$(python3 -c "import torch_npu,os;print(os.path.dirname(torch_npu.__file__))")/lib:$LD_LIBRARY_PATH

Check the driver and Python packages before loading the model:

npu-smi info
python3 -c "import torch, torch_npu; print(torch_npu.npu.is_available(), torch_npu.npu.device_count())"

If npu-smi reports a missing driver library, add the corresponding driver lib64 directories to LD_LIBRARY_PATH according to the host installation.

Launch the server

vllm serve "${MODEL_PATH}" \
    --served-model-name MiniCPM5-2B \
    --dtype bfloat16 \
    --max-model-len ${CTX_LEN:-32768} \
    --gpu-memory-utilization ${MEM_FRAC:-0.85} \
    --tensor-parallel-size ${TP_SIZE:-1} \
    --trust-remote-code \
    --port ${PORT:-8000}

Set TP_SIZE to the number of devices in ASCEND_RT_VISIBLE_DEVICES when using tensor parallelism.

Validate chat completions

MiniCPM5-2B is think-only. Use temperature=1.0, top_p=0.95, and enable_thinking=true:

curl http://localhost:${PORT:-8000}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-2B",
        "messages": [{"role": "user", "content": "1+1=?"}],
        "temperature": 1.0,
        "top_p": 0.95,
        "max_tokens": 256,
        "chat_template_kwargs": {"enable_thinking": true}
    }'

Expect HTTP 200 and an answer containing 2 in choices[0].message.content. Leave enough max_tokens for the model's reasoning text.

Tool calling

MiniCPM5-2B emits XML-style tool calls. This section requires matching vLLM and vLLM-Ascend 0.23.0; that release includes the minicpm5 parser in the OpenAI server layer, so no parser plugin file is needed:

pip install vllm==0.23.0
pip install vllm-ascend==0.23.0 \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi

vllm serve "${MODEL_PATH}" \
    --served-model-name MiniCPM5-2B \
    --dtype bfloat16 \
    --max-model-len ${CTX_LEN:-32768} \
    --gpu-memory-utilization ${MEM_FRAC:-0.85} \
    --tensor-parallel-size ${TP_SIZE:-1} \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser minicpm5 \
    --port ${PORT:-8000}

When sending tools, use both server flags above and the same 2B sampling settings:

{
  "model": "MiniCPM5-2B",
  "messages": [{"role": "user", "content": "What is the weather in Beijing?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get current weather for a city",
      "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"]
      }
    }
  }],
  "tool_choice": "auto",
  "temperature": 1.0,
  "top_p": 0.95,
  "max_tokens": 512,
  "chat_template_kwargs": {"enable_thinking": true}
}

Common failures

• torch_npu cannot import: source the CANN/NNAL environment and check that the installed PyTorch, TorchNPU, CANN, vLLM, and vLLM-Ascend versions match the supported stack.
• No devices are visible: check npu-smi info, ASCEND_RT_VISIBLE_DEVICES, and the Docker device/driver mounts. Do not use CUDA_VISIBLE_DEVICES for this backend.
• HBM allocation or 128K startup failure: lower --max-model-len, then lower --gpu-memory-utilization.
• Tensor-parallel initialization failure: make --tensor-parallel-size match the number of visible Ascend devices.
• Tool requests fail with an auto-tool-choice error: launch with both --enable-auto-tool-choice and --tool-call-parser minicpm5.

Reference

See docs/deployment/vllm-ascend.md for the human-readable cookbook and the official vLLM-Ascend installation guide for hardware-specific setup.