majiayu000 py / 14 / 15
General SOP for common requests related to py, 14, 15.
Prompt
Follow this SOP (replace specifics with placeholders like //):
- RuntimeError: CUDA driver error: invalid argument
- 路径：`vllm/distributed/device_communicators/symm_mem.py
- torch_symm_mem.rendezvous
- 这类报错通常是 “驱动/硬件/拓扑/权限不满足 symmetric memory 要求” 导致的，而不是模型权重或dtype的问题
- 下面给你一个“最可能原因
- 验证方法
- 解决办法”的梳理
- 1）最可能原因：GPU/驱动不支持（或不完全支持）PyTorch symmetric memory / NVSHMEM 类能力
- vLLM 0.11.0 默认可能会启用一种更激进的 GPU 通信（
SymmMemCommunicator），它依赖 PyTorch 里_symmetric_memory的能力；而这个能力对以下条件很敏感 - GPU 架构**（例如部分非 H100/非新架构可能不支持或支持不完善
For each step, include: action, checks, and failure rollback/fallback plan. Output format: for each step number, provide status/result and what to do next.
Triggers
- Use when the user asks for a process or checklist.
- Use when you want to reuse a previously mentioned method/SOP.
Examples
Example 1
Input:
Break this into best-practice, executable steps.