Nvidia Nixl

NVIDIA Inference Xfer Library (NIXL)

Target audience: operators wiring NIXL into Dynamo/vLLM/SGLang clusters, plugin authors writing new backends, developers using the agent API directly from Python (references/python-api.md). C++/Rust developers: consult src/api/cpp/ headers and examples/{cpp,rust}/ upstream directly — this skill does not carry a C++/Rust API reference. Assumes datacenter-class GPUs (H100/H200/B200/B300) with NVIDIA driver, CUDA 12.8+, RDMA NIC (Mellanox/EFA) for cross-node, and Linux (Ubuntu 22.04/24.04 or Fedora). macOS and Windows are not supported.

What NIXL is — one paragraph

NIXL is the transport, not a cache. If the goal is a KV cache that outlives a single vLLM process, the thing being configured is lmcache-mp (same inference-cache plugin) — a standalone LMCache server that can use NIXL as one of its backends. Come here for the agent API, plugins, and wire-level behaviour; go there for the cache server's deployment, sizing, and ZMQ wiring.

NIXL is a thin abstraction over heterogeneous transport backends. A nixlAgent registers memory regions (DRAM, VRAM, FILE, BLOCK, OBJ), exchanges metadata with peer agents via either ETCD or socket side-channel, then issues asynchronous one-sided READ/WRITE transfers between local and remote registered memory. The agent picks the best backend (UCX for network, GDS for storage, etc.) based on memory types and what both sides have loaded. Same-process loopback, intra-node GPU-to-GPU, and cross-node RDMA are all the same API. Two operations only — read and write — and both are non-blocking with optional notifications.

Version snapshot — verify before recommending

For staleness — see references/sources.md for verified URLs. Run freshen mode of skill-improver to refresh.

Decision tree — picking a backend

Is the transfer across nodes?
├─ Yes → Network backend
│   ├─ Standard RDMA (RoCE / IB) on Mellanox NICs?      → UCX
│   ├─ AWS EFA?                                          → libfabric (validated) or UCX
│   ├─ Heterogeneous GPU/NIC, software transport (no RDMA)? → UCCL [Preview]
│   ├─ KVCache-centric workload, multi-protocol (TCP/RDMA/CXL/NVMe-oF)? → mooncake [Preview]
│   └─ GPU-driven RDMA (GDAKI, kernel-launched)?        → gpunetio (DOCA)
└─ No → Local / storage backend
    ├─ NVMe / parallel FS via GPUDirect Storage?         → cuda_gds (single-thread) or gds_mt (multi-thread)
    ├─ Plain POSIX file (libaio default, liburing opt)?  → posix
    ├─ DeepSeek 3FS distributed FS?                      → hf3fs
    ├─ Block storage via GUSLI shared-mem client?        → gusli
    ├─ S3 (or S3-compatible) object store?               → obj (with optional cuobjclient accelerated engine)
    ├─ DDN Infinia storage?                              → infinia (v1.3.0+)
    └─ Azure Blob?                                       → azure_blob

The published manylinux wheels bundle libplugin_INFINIA.so by default as of v1.3.1 (#1832); the proprietary DDN libred_* runtime libraries are deliberately not vendored and are loaded from the customer's DDN install at /opt/ddn/red.

A single agent can instantiate multiple backends; per-transfer the agent chooses one based on the memory types involved and what the remote side advertises. Pass backends=["UCX","GDS"] to nixl_agent_config (Python) or createBackend calls (C++) to constrain candidates.

Quick start (Python)

import torch
from nixl import nixl_agent, nixl_agent_config

# Agent on each side
config = nixl_agent_config(
    enable_prog_thread=True,
    enable_listen_thread=True,        # socket side-channel
    listen_port=5555,
    capture_telemetry=False,
    backends=["UCX"],                  # default; use ["UCX","GDS"] etc. to add more
)
agent = nixl_agent("agent-1", config)

# Register a tensor (auto-detects DRAM vs VRAM)
tensor = torch.zeros((10, 16), dtype=torch.float32)
reg = agent.register_memory(tensor)

# After exchanging metadata (see references/python-api.md):
local_descs = agent.get_xfer_descs([tensor[i, :] for i in range(10)])
xfer = agent.initialize_xfer("READ", local_descs, target_descs, "agent-2", b"done")
agent.transfer(xfer)
while agent.check_xfer_state(xfer) == "PROC":
    pass  # spin or do other work

End-to-end working programs in examples/python/basic_two_peers.py, expanded_two_peers.py, partial_md_example.py. Full surface in references/python-api.md.

Metadata exchange — choose ONE of two modes

Side-channel (default). Each agent runs a TCP listener (enable_listen_thread=True, listen_port=N). One agent calls fetch_remote_metadata(remote_name, ip, port) to pull, or send_local_metadata(ip, port) to push. Good for fixed-pair setups, lab environments. Defaults to port 5555.

Since v1.2.0 (#1439) you can pass listen_port=0 to let the OS assign a free port; the bound port is recovered via getsockname() and logged at NIXL_INFO. Use this when several agents share a host and would otherwise collide on a hard-coded port — the peer then needs the port out-of-band. The same release widened the port fields (listenPort / listen_port) from int to uint16_t in nixl_params.h / nixl_types.h, and the Rust bindings gained a DEFAULT_COMM_PORT constant.

ETCD. Set NIXL_ETCD_ENDPOINTS=http://etcd:2379 (comma-separated for HA). Each agent calls sendLocalMD() / fetchRemoteMD(remote_name) (no IP/port args). Required for elastic / dynamic-scaling clusters where peers are not known upfront. ETCD is also how nixlbench discovers workers.

Both modes support send_partial_agent_metadata(descs, inc_conn_info, backends, label=...) — only register-then-send the metadata for specific descriptor lists, useful when memory regions are dynamic or to avoid advertising everything. Example: examples/python/partial_md_example.py.

Plugin Manager + plugin search path

The plugin manager defers loading until first use (#1546, in v1.0.0+) and reads NIXL_PLUGIN_DIR to find dynamic plugins on disk. Static plugins are compiled in. Set this env var explicitly when running from non-system paths:

export NIXL_PLUGIN_DIR=/path/to/nixl/lib/x86_64-linux-gnu/plugins

agent.getAvailPlugins() lists what was discovered. agent.getPluginParams("UCX") returns the param schema + supported memory types for that plugin (use this to discover what to pass to createBackend).

NIXL-EP — elastic Expert Parallel device kernels

examples/device/ep/csrc/ ships device-side CUDA kernels for MoE all-to-all dispatch — nixl_ep_ll.cu (low-latency) and nixl_ep_ht.cu (high-throughput). Two API surfaces, one mode per agent: mixing LL and HT calls on the same agent is a hard error (mode guards added in v1.0.1, #1538). NIXL-EP also supports elastic scale-up (new nodes joining a running deploy), with signaling-buffer fixes in v1.0.1 (#1453). GPU timeouts are configurable (#1520). NIXL-EP is the layer Dynamo's MoE plane will land on; for plain disaggregated prefill (single tensor transfer), use the regular agent API.

Breaking change in v1.3.0 (#1693): rank/expert semantics were refactored for elastic rank handling — dispatch/combine now take an active-rank bound plus experts-per-rank parameterization, buffers/layouts moved to an active-range model, mask updates became public with host-side active-rank tracking, and the legacy mask-clean API was removed. Code written against 1.2.x NIXL-EP needs porting. vLLM has been consuming this: NIXL EP + DBO (#45275), elastic-EP communicator (#45013), and NVFP4 post-receive quantization skip (#45606) all landed in vLLM v0.24.0.

Telemetry — two exporters, environment-driven

Cyclic buffer = static plugin, shared-memory ring; readers in examples/python/telemetry_reader.py and examples/cpp/telemetry_reader.cpp. Prometheus exporter is dynamic + experimental (beta) as of v1.0.x — see src/plugins/telemetry/prometheus/README.md. The Prometheus Exposer is shared across agents in the same process (v1.0.0 PR #1470). Built-in metrics: agent_tx_bytes, agent_rx_bytes, agent_xfer_time (µs), agent_xfer_post_time (µs), agent_memory_registered, etc. — full table in references/architecture.md and per-transfer telemetry via agent.get_xfer_telemetry(handle).

Common gotchas — read before deploying

1. UCX_TLS is the #1 source of segfaults in vLLM+NIXL deploys. UCX_TLS=tcp alone segfaults nixlUcxSharedThread::run() after prefill on CUDA-capable images that haven't been told the GPU is reachable. Use UCX_TLS=cuda_copy,sm,tcp (or full cuda_copy,cuda_ipc,sm,tcp,rc for cross-node). Symptom: W ucx_utils.cpp:581: memory is detected as host. (Verified in vllm-caching skill 2026-04-25.)
2. First transfer after pod-ready may fail with NIXL_ERR_REMOTE_DISCONNECT — handshake race. Retry the second request. Side-channel listener takes a few seconds to bind.
3. Side-channel host MUST be a real interface, not a service VIP. In Kubernetes, set VLLM_NIXL_SIDE_CHANNEL_HOST=$(POD_IP) via downward API and use a headless Service (clusterIP: None, publishNotReadyAddresses: true). Same applies to ETCD-mode if pods advertise their own addr.
4. NIXL_PLUGIN_DIR defaults aren't reliable in non-pip installs. If agent.getAvailPlugins() returns [] and an exception fires (No plugins available for NIXL), the env var is wrong. Verify with ls $NIXL_PLUGIN_DIR/libplugin_*.so.
5. Mooncake plugin disables progress thread, has its own metadata system that bypasses NIXL's, and caps transfer requests per handle at 1024 (kMaxRequestCount). It's [Preview]. Don't compose with strict NIXL-only metadata flows.
6. GDS requires cufile.json with allow_compat_mode: true unless full GDS is wired (kernel module + supported FS). The plugin README has the canonical config and CUFILE_ENV_PATH_JSON export.
7. HF3FS needs page-aligned, page-size-multiple memory for the zero-copy mmap() shared-memory path; otherwise it copies. Pass mem_config=dram_zc to fail loud if alignment is wrong.
8. gpunetio (DOCA) is single-NIC + single-GPU per backend. To use 2 NICs, instantiate 2 backends with different network_devices. nvshmem-aware bench mode supports VRAM-only transfers.
9. EFA-only configs in UCX were gated to not poison non-EFA setups in v1.0.1 (#1527). Below that, UCX with EFA-tuned defaults could degrade other systems.
10. Telemetry timestamps were removed from events in v1.0.0 (#1522) — readers must derive ordering from ring-insertion order, not event timestamp fields. The category field was also removed in v1.3.0 (#1649); consumers parsing telemetry events must drop it.
11. Path-mode FILE_SEG registrations need a unique devId each (v1.3.1, #1790). v1.3.0 added path-based file registration — declare a file by path in nixlBlobDesc::metaInfo as <modes>:<path> (ro/rw plus direct/sync/noatime/create), and POSIX / HF3FS / CUDA_GDS / GDS_MT open it in registerMem and close it in deregisterMem. Reusing an in-use devId across distinct path-mode files caused a double-free on deregister and is now rejected with NIXL_ERR_INVALID_PARAM. The older fd-in-devId mode is unchanged — one fd may still back multiple descriptors at different offsets.
12. Building from source requires a C++20 toolchain since v1.3.0 (#1571). Full-sweep CUDA builds are slow; pass -Dnixl_cuda_arch_list=90,100 (or your SM list) instead of the default sm_80,86,89,90,100,103,120.

Full debugging cookbook in references/gotchas.md.

What this skill does NOT cover (and where to go)

• vLLM NixlConnector configuration (--kv-transfer-config, kv_role, K8s pod shape, the proxy server) — covered in vllm-caching skill, especially references/connectors.md "NixlConnector" section. The vllm-caching skill has live-lab-verified recipes for 1P1D Qwen3-4B on consumer hardware including the six non-obvious env vars.
• Dynamo deployment and disaggregation orchestration — Dynamo docs at https://docs.nvidia.com/dynamo/. NIXL is the data plane; Dynamo handles request routing, scheduling, and the rest of the control plane.
• vLLM-stack production deploy (LMCache, MooncakeConnector etc.) — vllm-deployment and vllm-caching skills.
• SGLang KV transport — SGLang has its own connector; NIXL is one of the available backends. Consult SGLang docs.

When the user is stuck — diagnostic flow

Quick first pass: run python ${CLAUDE_SKILL_DIR}/scripts/check_install.py --backends UCX,LIBFABRIC,GDS (substitute backends used). Validates wheel/CUDA match, plugin discovery, UCX_TLS, ETCD reachability, cufile.json — covers gotchas #1, #2, #4, #5, #7 in references/gotchas.md.

1. nixl_agent raises RuntimeError("No plugins available") → NIXL_PLUGIN_DIR is wrong, OR pip wheel mismatch (nixl-cu13 on CUDA 12 host etc.). Confirm pip show nixl-cu12 nixl-cu13 and which one PyTorch CUDA matches.
2. Hangs on fetch_remote_metadata → side-channel listener didn't start. Check enable_listen_thread=True on the target, port not collided, firewall open. Add a 5 s sleep before fetching.
3. NIXL_ERR_REMOTE_DISCONNECT → handshake race or network ACL. Retry. Check UCX_TLS. Confirm peer is listening with ss -tlnp | grep <port>.
4. Posting transfer failed. (state == "ERR") → the agent rejected the request. Common causes: descriptor not within registered region, remote agent metadata not loaded yet (call check_remote_metadata(name) first), or backend not available on both sides. Check agent.query_xfer_backend(handle) if it's prepped.
5. Throughput nowhere near line rate → run nixlbench --backend UCX --etcd-endpoints ... --initiator_seg_type=VRAM --target_seg_type=VRAM to isolate. Compare against AWS/Mellanox-published numbers. Check UCX_TLS, UCX_NET_DEVICES, cudaDeviceCanAccessPeer, GDS path with gdscheck.
6. Correctness bug after upgrade → git log v1.0.0..v1.0.1 and search for the relevant subsystem; the libfabric / NIXL-EP subsystems had several correctness fixes in 1.0.1.

references/gotchas.md has the full diagnostic flow with commands.

References

• references/architecture.md — Agent + Memory Section + South Bound API (SB API) + Plugin Manager + descriptor lists + NIXL-EP device kernels + telemetry event catalog.
• references/plugins.md — All plugins with deps, parameters, capabilities, when-to-pick (13 documented in depth; infinia and tracing added in the 1.3 line — read their src/plugins/<name>/README.md directly).
• references/python-api.md — nixl_agent Python surface with worked examples for every common operation.
• references/deployment.md — pip install, source build (meson+ninja), Docker, K8s, ETCD setup, env vars, nixlbench, kvbench.
• references/integrations.md — Dynamo, vLLM NixlConnector, SGLang, observability stack pointers.
• references/gotchas.md — Debugging cookbook, error catalog, threading/sync model, build pitfalls.
• references/sources.md — Verified URLs with Last verified dates.
• scripts/check_install.py — Sanity-check script: wheel/CUDA match, plugin discovery, UCX_TLS, ETCD reach, cufile.json. Run before debugging deeper.

Authoritative upstream sources

Primary: https://github.com/ai-dynamo/nixl (releases via gh release list --repo ai-dynamo/nixl). Canonical docs live in-tree under docs/ (nixl.md, BackendGuide.md, telemetry.md, python_api.md); Python API source src/api/python/_api.py; per-plugin docs src/plugins/<name>/README.md; examples examples/{python,cpp,rust,device/ep}/; benchmarks benchmark/{nixlbench,kvbench}/. Full URL list with verification dates in references/sources.md.

When asked about a specific backend, read the plugin's README.md under src/plugins/<name>/ first — every plugin documents its own deps, params, and gotchas there. Then map to references/plugins.md for the cross-cutting matrix.