Known Issues

MORI Known Issues

Environment-level problems that look like MORI bugs but are not. Check here before debugging MORI itself when performance is unexpectedly bad on a specific machine.

Issue 1 — VMM peer traffic falls off XGMI onto PCIe / host memory

Affects: mori-cco (EPv2, dispatch_combine_v2) and anything else built on the HIP VMM API. Does not affect mori-shmem (EPv1), which uses IPC handles.

Root cause: the Linux kernel was compiled without CONFIG_DMABUF_MOVE_NOTIFY=y (and usually also without CONFIG_PCI_P2PDMA=y). This is the case for the stock Ubuntu 22.04 GA 5.15 kernel. It is a kernel build configuration problem, not a kernel version bug and not an amdgpu regression — a 5.15 rebuilt with those options works fine, and the Ubuntu 6.8 HWE kernel already has them.

Impact is silent. Nothing errors, correctness still passes; the a2a path is just ~9x slower.

Symptoms

• EPv2 dispatch/combine bandwidth is a fraction of EPv1's on the same machine.
• Per-rank bandwidth plateaus at roughly a single XGMI link (~55 GB/s on MI355X), or below it — even though the box has 7 peers.
• Aggregate bandwidth does not increase with the number of peers: 1 peer and 7 peers give the same total.
• rocm-smi --showtopotype shows XGMI everywhere and TransferBench reports full aggregate bandwidth — because those use the non-VMM path and are unaffected.

Measured on an 8x MI355X box (HIDDEN=7168 TOPK=8 EPR=32, bf16, 4096 tok/rank, 8 ranks):

Detection (30 seconds, no GPU needed)

grep -E 'CONFIG_(PCI_P2PDMA|DMABUF_MOVE_NOTIFY)=' /boot/config-$(uname -r)

Both must print =y. If either is missing (# CONFIG_... is not set or no output), the machine is affected.

Also useful:

uname -r                                    # 5.15.0-*-generic (Ubuntu GA) is the usual offender
grep -E 'CONFIG_IOMMU_DEFAULT' /boot/config-$(uname -r)

Runtime confirmation

vmm_peer_probe.cpp in this skill directory gives a direct verdict — it allocates VMM memory on one GPU, grants peer access from another, and checks whether the memory stayed in VRAM:

hipcc -O3 --offload-arch=$(rocminfo | grep -m1 -o 'gfx[0-9a-f]*') \
      vmm_peer_probe.cpp -o vmm_peer_probe
./vmm_peer_probe            # defaults: owner=1 peer=0 size=8GB

• VERDICT: OK — memory stayed in VRAM, the machine is fine.
• VERDICT: AFFECTED — granting peer access evicted the buffer out of VRAM.

Mechanism (what actually happens)

1. hipMemSetAccess() → ROCr hsa_amd_vmem_set_access() imports the allocation into each peer as a dma-buf (ImportMemoryHandle(DMABUF_FD) → hsaKmtHandleImport → KFD import_obj_create() → amdgpu_gem_prime_import() → dma_buf_dynamic_attach()).

2. The dma-buf core then pins the exporter's buffer, calling amdgpu_dma_buf_pin() (amd/amdgpu/amdgpu_dma_buf.c):

   u32 domains = bo->allowed_domains;              /* VRAM | GTT */
   if (!IS_ENABLED(CONFIG_DMABUF_MOVE_NOTIFY)) {
           domains &= ~AMDGPU_GEM_DOMAIN_VRAM;     /* <-- VRAM stripped */
   } else {
           list_for_each_entry(attach, &dmabuf->attachments, node)
                   if (!attach->peer2peer)
                           domains &= ~AMDGPU_GEM_DOMAIN_VRAM;
   }
   return amdgpu_bo_pin(bo, domains);              /* GTT = host system memory */
   

   IS_ENABLED() is a compile-time constant taken from the kernel's own autoconf.h. Without move-notify support the kernel cannot ask an importer to drop its mapping when a buffer must move, so amdgpu conservatively refuses to pin in VRAM.

3. The buffer is therefore pinned into host system memory. Note this is stronger than "peer access falls back to PCIe": the data is not in peer VRAM at all. Every subsequent "peer" store is a write to host DRAM across the writing GPU's own PCIe link, which is why aggregate bandwidth is capped at ~one PCIe link no matter how many peers there are.

4. amdgpu_dma_buf_map() contains a peer2peer/VRAM placement check, but it is guarded by if (!bo->tbo.pin_count) and the pin already happened — that code never runs.

Why EPv1 is immune: hsa_amd_agents_allow_access() maps the same BO into every GPU's page tables in one hsaKmtMapMemoryToGPUNodes() call. No dma-buf, no attach, no pin.

Why there is no application-level workaround: MORI runs one process per rank, so peer memory must cross process boundaries, which requires exportable handles — and ROCr unconditionally does the per-peer dma-buf import (it even lazily exports a dma-buf fd for locally created handles). cco also cannot fall back to IPC handles because it needs a flat symmetric VA at computed addresses, which only hipMemAddressReserve() + hipMemMap() can provide.

Fix

Preferred — install a kernel that already has the options (Ubuntu 22.04):

sudo apt install linux-generic-hwe-22.04       # 6.8; amdgpu DKMS rebuilds automatically
sudo reboot

If the machine must stay on 5.15 — rebuild it with two options. All their Kconfig dependencies (ZONE_DEVICE, MEMORY_HOTPLUG, SPARSEMEM_VMEMMAP, GENERIC_ALLOCATOR, DMA_SHARED_BUFFER, …) are already satisfied in the stock config:

sudo apt build-dep linux-image-unsigned-$(uname -r)
apt-get source linux-image-unsigned-$(uname -r)
cd linux-5.15.0-*/
cp /boot/config-$(uname -r) .config
scripts/config -e DMABUF_MOVE_NOTIFY -e PCI_P2PDMA
scripts/config -d DEBUG_INFO -d DEBUG_INFO_BTF     # much faster build, much smaller packages
make olddefconfig
grep -E 'CONFIG_(DMABUF_MOVE_NOTIFY|PCI_P2PDMA)=' .config   # verify before building
make -j$(nproc) bindeb-pkg
sudo dpkg -i ../linux-image-*.deb ../linux-headers-*.deb
sudo reboot                                         # amdgpu DKMS rebuilds for the new kernel

Check mokutil --sb-state first — with Secure Boot enabled a self-built kernel needs signing. The old kernel stays in GRUB, so this is revertible. Note that a self-built kernel leaves the distro's kernel maintenance path; prefer the HWE kernel for anything long-lived.

CONFIG_DMABUF_MOVE_NOTIFY is the one that actually matters. CONFIG_PCI_P2PDMA is enabled alongside it because upstream HSA_AMD_P2P depends on both and it costs nothing.

Third option — one-line amdgpu DKMS patch, no kernel change. Force the branch that keeps VRAM (its per-attachment peer2peer check is already satisfied):

S=/usr/src/amdgpu-<version>/amd/amdgpu/amdgpu_dma_buf.c
sudo cp $S $S.orig
sudo sed -i 's|if (!IS_ENABLED(CONFIG_DMABUF_MOVE_NOTIFY)) {|if (0) { /* WAR: kernel lacks MOVE_NOTIFY */|' $S
sudo dkms build  amdgpu/<version> -k $(uname -r) --force
sudo dkms install amdgpu/<version> -k $(uname -r) --force

Then either reboot, or reload the module (modprobe -r amdgpu && modprobe amdgpu) if nothing holds the GPUs (lsmod | grep '^amdgpu' must show used_by=0 and rocm-smi --showpids must show no KFD PIDs). Reloading invalidates every container's GPU context, so it disturbs other users exactly as much as a reboot — it only saves time. Roll back with sudo cp $S.orig $S and rebuild.

Verification after the fix

grep -E 'CONFIG_(PCI_P2PDMA|DMABUF_MOVE_NOTIFY)=' /boot/config-$(uname -r)   # both =y
./vmm_peer_probe                                                             # VERDICT: OK

Then re-run the EPv2 kernel bench; per-rank bandwidth should be in the same range as EPv1.

About IOMMU — related but not the cause

amd_iommu=on iommu=pt does not fix Issue 1. This was tested directly: with passthrough confirmed active (dmesg showing iommu: Default domain type: Passthrough, and /sys/bus/pci/devices/<BDF>/iommu_group/type reading identity), bandwidth and IOMMU transaction counters were unchanged. The buffer is pinned into host memory before any address-translation question arises.

The tempting but wrong theory: IOMMU passthrough sets adev->ram_is_direct_mapped, which KFD's reuse_dmamap() uses to choose KFD_MEM_ATT_SHARED over KFD_MEM_ATT_DMABUF. That choice really does flip with iommu=pt — but both attachment types share the same scatter-gather BO, whose pages were already placed in host memory by the pin. Attachment type is not the physical route.

Still enable iommu=pt anyway. It is the standard configuration for AMD GPU / HPC nodes, amdgpu uses ram_is_direct_mapped elsewhere, and it removes IOMMU translation overhead. Just do not let it stand in for the kernel config fix. Note that Ubuntu kernels default to translated mode (CONFIG_IOMMU_DEFAULT_DMA_LAZY=y, CONFIG_IOMMU_DEFAULT_PASSTHROUGH not set), so passthrough must be requested explicitly on the kernel command line.

IOMMU is, however, an excellent diagnostic instrument for this issue, because host-DRAM DMA is translated by the IOMMU while XGMI peer traffic never touches it:

EV=""; for i in 0 1 2 3 4 5 6 7; do EV="$EV -e amd_iommu_$i/mem_trans_total/"; done
sudo perf stat -a $EV -- sleep 8      # while an a2a / peer-write workload runs

On an affected machine one IOMMU counts billions of transactions during VMM peer traffic (64 B each — multiply out and it matches the observed bandwidth exactly). After the fix the same measurement drops by ~380x, down to the same noise level as non-VMM peer traffic. amd-smi metric --pcie is a coarser cross-check: on an affected box exactly one card — the one issuing the writes — shows sustained multi-GB/s PCIe traffic while all its peers stay idle, which is itself the tell that the peers are not the destination.

Also worth knowing

• CONFIG_HSA_AMD_P2P cannot be relied on. amd/dkms/dkms-config.sh tests for the misspelled CONFIG_DMABUF_MOVENOTIFY (missing underscore), so the amdgpu DKMS build derives CONFIG_HSA_AMD_P2P=0 even on kernels where both real options are =y. Present in at least amdgpu DKMS 6.14.14 and 6.16.13. Do not use CONFIG_HSA_AMD_P2P as a health signal, and do not expect code under #ifdef CONFIG_HSA_AMD_P2P to be compiled in.

• KFD may report max_bandwidth=0 for XGMI io-links (/sys/class/kfd/kfd/topology/nodes/*/io_links/*/properties) on some hosts while others report 64000, with no functional difference. It is unexplained but unrelated to Issue 1 — kfd_mem_attach() never consults link bandwidth. Do not chase it.

• Micro-benchmarking peer bandwidth is noisy. Use ≥1024 blocks per stream and take the best of several repetitions. An under-parallelised harness produced a 1.5x spread between consecutive identical runs, which is more than enough to invent or hide a regression.

Issue 2 — ROCm 7.0–7.2.x: hipMemSetAccess sub-buffer bug (cco 2-node cannot start)

Affects: mori-cco (EPv2) multi-node. A clr/HIP-runtime bug present in all ROCm 7.0 → 7.2.x releases (regression from 6.4.x; fixed on clr develop 2026-01-26 but not back-ported to any 7.2.x release branch). ROCm 7.14+ already has the fix. mori-shmem (EPv1) is unaffected.

Symptom: ccoDevCommCreate fails during window setup:

[shmem] [error] ccoMemAlloc: hipMemSetAccess failed after retries: 1 (invalid argument)
[shmem] [error] ccoDevCommCreate: resource window MemAlloc failed
RuntimeError: ccoDevCommCreate failed: -1

Single-node cco usually still passes; the failure shows up cross-node. CCO_GDR_CAPABLE=0 does not help — the trigger is sub-buffer geometry, not GPUDirect-RDMA.

Root cause: hipMemSetAccess validates coverage by walking the parent reservation's sub-buffers from sub-buffer 0, ignoring the ptr argument, and returns InvalidValue whenever size is not a prefix-sum of sub-buffer sizes starting at sub-buffer 0. cco reserves one flat VA (hipMemAddressReserve) and maps several different-sized windows (sub-buffers) into it, each with its own hipMemSetAccess, so every window after the first fails. (SWDEV-568260 / ROCm/rocm-systems#2451, commit be8bcd059.)

Fix — upgrade, or rebuild clr with the upstream fix and swap libamdhip64.so.

Prefer ROCm 7.14+. If you must stay on 7.2.x, rebuild clr for your exact version and cherry-pick the fix (this is what docker/Dockerfile.cco did before the 7.14 move — git log -S be8bcd059 -- docker/ for the original recipe):

V=$(cat /opt/rocm/.info/version | head -c5)          # e.g. 7.2.3
git clone --depth 1 --branch rocm-$V https://github.com/ROCm/clr.git /tmp/clr
git clone --depth 1 --branch rocm-$V https://github.com/ROCm/HIP.git /tmp/hip
cd /tmp/clr && git fetch origin develop && git cherry-pick be8bcd059     # clean on 7.2.x
cmake -B build -DHIP_COMMON_DIR=/tmp/hip -DCMAKE_PREFIX_PATH=/opt/rocm \
      -DCLR_BUILD_HIP=ON -DCLR_BUILD_OCL=OFF -DHIP_PLATFORM=amd \
      -D__HIP_ENABLE_RTC=OFF -D__HIP_ENABLE_PCH=OFF
make -C build amdhip64 -j$(nproc)
# back up and replace the real file behind the libamdhip64.so.7 symlink, keeping its name:
T=$(readlink -f /opt/rocm/lib/libamdhip64.so.7)
cp "$T" "$T.orig" && cp build/hipamd/lib/libamdhip64.so.*-* "$T"

The rebuilt lib links the system libhsa-runtime64.so.1 (ROCR is not touched). Deploy by replacing the file (or LD_PRELOAD) and restarting the process; it only works on the exact matching ROCm version (soname must match, e.g. libamdhip64.so.7.2.70203 for 7.2.3).

Issue 3 — ROCm 7.2.x: uncached VMM allocations land in the coarse-grained pool (cco combine slow)

Affects: mori-cco (EPv2) combine latency on ROCm 7.2.x (fixed on clr develop, not in 7.2.x release; 7.14+ is fine). Correctness is unaffected — this is a pure perf issue, and it usually rides along with Issue 2 on the same 7.2.x box.

Symptom: cco combine is ~10–15 µs slower than on ROCm 7.14 for the same shape, and requesting uncached memory (the default) is no faster than forcing cached (CCO_UNCACHED_WINDOW=0 gives the same time). Example: v1_ll combine, tok16, hidden 6144, fp8/bf16, EP16 → ~65 µs on 7.2.x vs ~54 µs on 7.14; dispatch is roughly unchanged.

Root cause: rocclr's VMM path (deviceVmemAlloc) always allocates from the coarse-grained gpuvm_segment_, ignoring hipMemAllocationTypeUncached. cco's symmetric window therefore stays cached, so a GPU spinning on an RDMA "done" flag hits a dirty L2 line and waits for coarse-grained coherence. The non-VMM path (deviceLocalAlloc / hipMalloc, which mori-shmem uses via hipDeviceMallocUncached) already routes uncached requests to the extended fine-grained pool; only the VMM path missed it.

Fix — upgrade to ROCm 7.14+, or add the same routing to the clr rebuild from Issue 2 (this is the upstream develop behavior): in rocclr/device/rocm/rocmemory.cpp propagate the uncached flag into deviceVmemAlloc (it is hard-coded to 0), and in rocclr/device/rocm/rocdevice.cpp Device::deviceVmemAlloc select gpu_ext_fine_grained_segment_ when the uncached flag is set and that pool exists, else fall back to gpuvm_segment_ (unchanged for every non-uncached allocation). Both Issue 2 and Issue 3 ship in a single rebuilt libamdhip64.so.