DOCA GPUNetIO ib_write_lat
Where to start: This is a tool skill for the GPUNetIO-
flavored ib_write_lat benchmark shipped under
doca/tools/gpunetio_ib_write_lat/ (a client + server pair,
built from source against the installed DOCA via meson).
It measures the latency of an RDMA WRITE work request when
the WR is posted from a CUDA kernel through the
doca-gpunetio device-side surface, in a ping-pong cadence.
Open TASKS.md and start at
## configure for the GPU-NIC pairing
precondition and the build pattern; jump to
## run for the single-iteration smoke
flow. Open CAPABILITIES.md when the
question is what this tool actually measures, how it
differs from the GPI sister tool on the same physical
operation, or how to interpret the half-iter / full-iter
/ CUDA-side usec output and the median / p99 / jitter
characterization. If DOCA is not installed yet, route to
doca-setup first; if the
user is still deciding between GPUNetIO and GPI as a
programming surface, the picture in
../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes
and
../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes
is the first stop.
Example questions this skill answers well
The CLASSES of doca-gpunetio-ib-write-lat questions this
skill is built to answer, each with one worked example. The
class is the load-bearing piece; the worked example is one
instance.
- "What GPU-init RDMA-WRITE latency / jitter can the
GPUNetIO path deliver for a real-time / control-loop
workload?" — worked example: "measure per-iteration
WRITE latency between two hosts with an H100 +
ConnectX-7 on each side, target the median and the p99
separately". Answered by the GPU-NIC pairing
precondition in
CAPABILITIES.md ## Capabilities and modes
- the bring-up flow in
TASKS.md ## configure +
TASKS.md ## run.
- "This is the GPUNetIO tool — how does the latency
number differ from the GPI programming surface?" —
worked example: "the team is using GPI; should I expect
GPUNetIO to beat / tie / lose vs GPI?". Answered by the
"same physical operation, different runtime framework"
rule in
CAPABILITIES.md ## Capabilities and modes
- the cross-link to the GPI library skill
../../libs/doca-gpi/CAPABILITIES.md
(note: doca/tools/ ships no GPI ib_write_lat
benchmark binary — GPI is a programming surface, not a
shipped benchmark tool).
- "Median vs p99 vs jitter — which one is the actual
answer for a real-time control loop?" — worked
example: "my control loop has a deadline; the median
is well under the budget but p99 spikes; do I quote
the median or the p99?". Answered by the
median-vs-p99-vs-jitter rule in
CAPABILITIES.md ## Observability
- the eval-loop overlay in
TASKS.md ## test.
- "What is the latency-vs-batching trade-off specific
to GPU-init RDMA?" — worked example: "my CUDA kernel
could batch multiple WRs to amortize the GPU-side
overhead; what does that buy me on latency vs what does
it cost me?". Answered by the
latency-vs-batching trade-off in
CAPABILITIES.md ## Capabilities and modes.
- "What version of DOCA + CUDA Toolkit do I need for
this binary to build and run?" — worked example: "my
install has DOCA at one semver and CUDA at another;
will the ToT-shipped
gpunetio_ib_write_lat even
link?". Answered by the version overlay in
CAPABILITIES.md ## Version compatibility.
- "How do I read the half-iter / full-iter / CUDA-side
usec columns?" — worked example: "the binary printed
half-iter, full-iter, and a CUDA-side number — what is
the right column to quote for one-way latency vs
round-trip vs cross-check?". Answered by the column-
semantics rule in
CAPABILITIES.md ## Observability.
Audience
This skill serves external developers and performance
engineers who need a reproducible measurement of the
latency of an RDMA WRITE WR when the WR is posted from a
CUDA kernel through doca-gpunetio, on the user's actual
install and GPU-NIC pair. Concretely:
- A developer designing a GPU-resident real-time control
loop and deciding whether the GPUNetIO path's tail
latency fits the deadline.
- A platform operator validating a tuning change (NUMA
pinning, GPU PCIe placement, IB device choice, GID
index, NIC firmware burn) by re-running this benchmark
against the new state.
- An SRE / performance engineer producing a "this is the
GPUNetIO-driven WRITE latency on this GPU-NIC pair
today, with median + p99 + jitter" artifact downstream
consumers can cite.
- An AI agent answering "is the doca-gpunetio latency
budget acceptable for this real-time workload class"
honestly — with measured numbers, the build +
invocation that produced them, and the GPU + NIC +
DOCA version that scopes them — rather than guessing.
It is not for users debugging the doca-gpunetio
library itself (route to
../../libs/doca-gpunetio/SKILL.md),
and not a substitute for the perftest upstream
ib_write_lat (which measures CPU-initiated WRITE
latency).
Language scope
The doca-gpunetio-ib-write-lat tool is shipped as C
plus CUDA .cu translation units under
doca/tools/gpunetio_ib_write_lat/, split into a
client/ subtree, a server/ subtree, and a common/
subtree shared between them (per the verified file layout:
client/{main.c,perftest.{c,h},meson.build},
server/{main.c,perftest.{c,h},meson.build},
common/{common.c,common.h,kernel.cu}). The host-side
build is meson against the installed DOCA pkg-config
modules (doca-gpunetio, doca-rdma, doca-common,
plus the CUDA Toolkit dependency); the device-side build
is nvcc against the DOCA GPU NetIO device-side header
set. There is no Python / Rust / Go binding — the tool is
a pair of CLI binaries.
When to load this skill
Load this skill when the user is — or the agent needs to
— build and run the gpunetio_ib_write_lat client +
server on real hosts with DOCA installed plus a CUDA
Toolkit matched to the DOCA install, and a GPU + IB device
pair on each host's PCIe topology. Concretely:
- Measuring kernel-initiated RDMA WRITE latency between
two hosts (or a host and a BlueField DPU) with the
GPUNetIO surface.
- Characterizing tail latency (p99 / p99.9) and jitter
for a real-time / control-loop workload class.
- Deciding whether the GPUNetIO path is the right runtime
surface for a class of workload vs the GPI programming
surface (the
doca-gpi
library — doca/tools/ ships no GPI benchmark binary)
or the classic CPU-initiated perftest path.
- Capturing a documented baseline (build + invocation +
DOCA version + GPU + NIC + as-deployed environment +
numbers) for later regression hunts.
- Diagnosing a build / link / run failure that surfaces
the GPUNetIO + RDMA bring-up sequence under this tool's
shipped scaffolding.
Do not load this skill for general DOCA orientation,
library API work, or installation. For those, use
doca-public-knowledge-map,
../../libs/doca-gpunetio/SKILL.md,
or doca-setup. Do not load
it for application-level real-time deadline analysis —
this benchmark measures the WR latency through GPUNetIO,
not the user's full pipeline.
What this skill provides
This is a thin loader. Substantive material lives in
two companion files:
CAPABILITIES.md — what the tool measures (the
ping-pong WRITE latency primitive driven by both sides'
CUDA kernels through doca-gpunetio), the
runtime-surface selection rule (GPUNetIO vs GPI vs
CPU-initiated), the GPU-NIC pairing precondition, the
latency-vs-batching trade-off intrinsic to GPU-init
RDMA, the median / p99 / jitter reporting taxonomy,
the version overlay (DOCA .pc PLUS CUDA Toolkit),
the layered error taxonomy, the observability surface
(stdout report including the timeout knob the
gpunetio_rdma_write_lat_* kernel functions surface
per the verified common.h), and the safety overlay.
TASKS.md — step-by-step workflows for the in-scope
task verbs: install, configure, build, modify,
run (smoke-before-bulk; single-iteration verification;
reading the report columns), test (the eval loop —
median / p99 / jitter / steady-state), debug (walk
the error taxonomy layer by layer), use (how a
latency result feeds a real-time class-of-workload
decision), plus a Deferred task verbs block.
The skill assumes a host where DOCA is already installed,
a CUDA Toolkit matched to the install is present, and the
operator has whatever privileges the public install
profile expects for binding a doca_dev, a doca_gpu,
and an OOB TCP socket.
What this skill deliberately does not ship
This skill is agent guidance, not a samples or
scripts bundle. It deliberately does not contain — and
pull requests should not add:
- Specific flag strings or expected latency numbers
beyond what the tool's shipped
--help and main.c
ARGP registration establish. The flag surface is small
(device name, GPU PCIe address, GID index, server IP
on the client side); the agent re-reads the binary's
--help on the installed version.
- Pre-written DOCA GPUNetIO or CUDA kernel source
code that would compete with the shipped tool tree.
The shipped
client/, server/, and common/
subtrees are the verified worked example.
- Wrappers, parsers, or scripts in any language that
consume the tool's stdout. The output format is small
and documented in
CAPABILITIES.md ## Observability.
- A
samples/, bindings/, or reference/ subtree.
This is a thin loader for a shipped tool tree.
Loading order
- Read this
SKILL.md first to confirm the user's
question is in scope (the user actually wants to
measure kernel-initiated WRITE latency through
GPUNetIO, not the GPI variant, not the CPU-initiated
variant, and not a library API question).
- For what the tool measures, the surface-selection
rule against the GPI sister tool and the CPU-initiated
perftest, the latency-vs-batching trade-off, the
median / p99 / jitter reporting taxonomy, the version
overlay, the error taxonomy, the observability
surface, and the safety overlay, see
CAPABILITIES.md.
- For step-by-step workflows —
install,
configure, build, modify, run, test,
debug, use — see TASKS.md.
Related skills
../../libs/doca-gpunetio/SKILL.md —
the library this tool wraps. The per-GPU doca_gpu
context, the GPU-visible RDMA handles, the CUDA-side
persistent-kernel pattern, the dual capability-
discovery rule (DOCA cap-query AND
cudaGetDeviceProperties), and the env preconditions
(nvidia_peermem loaded, CUDA buffers registered
with DOCA) live there.
../../libs/doca-rdma/SKILL.md —
the underlying RDMA library. The RDMA queue this tool
binds is created and connected via doca-rdma; the
queue lifecycle, the transport type (RC vs UC vs UD),
the permission matrix, and the connection method are
owned there.
../../libs/doca-verbs/SKILL.md —
the raw-verbs escape hatch beneath doca-rdma /
doca-gpunetio. This tool stays on the higher-level
surfaces.
../doca-gpunetio-ib-write-bw/SKILL.md —
bandwidth analog of this tool on the same runtime
framework. Same physical operation; different metric
class (latency vs BW). The two together carry the full
GPUNetIO-side latency / throughput picture.
doca-gpi — the GPI
programming surface (CUDA-kernel-initiated RDMA). The
alternative runtime framework for the same physical
operation; doca/tools/ ships no GPI ib_write_lat
benchmark binary, so the GPI comparison is against the
library surface, not a sibling tool. The selection rule
in
CAPABILITIES.md ## Capabilities and modes
is the decision aid; the agent's job is to teach when
to pick which.
doca-version — the
canonical version-detection chain, four-way match
rule. The ## Version compatibility section here is a
thin overlay.
doca-setup — env
preparation, install verification, GPU + CUDA Toolkit
pairing, nvidia_peermem load, hugepages, NUMA, and
the NGC DOCA container path.
doca-public-knowledge-map —
routing to the public DOCA documentation set and the
CUDA Toolkit pointer.
doca-debug — the
cross-cutting debug ladder.
doca-hardware-safety —
the bundle-wide hardware-safety meta-policy.
1---2license: Apache-2.03name: doca-gpunetio-ib-write-lat4description: >5 Use this skill when the user is measuring GPU-kernel-initiated RDMA6 WRITE latency through doca-gpunetio — building and running the7 `gpunetio_ib_write_lat` client + server pair under8 `doca/tools/gpunetio_ib_write_lat/`, checking GPU-NIC pairing,9 reading the half-iter / full-iter / CUDA-side usec columns,10 characterizing median / p99 / jitter for a real-time control loop,11 picking GPUNetIO vs GPI vs CPU-initiated `perftest`, or weighing the12 latency-vs-batching trade-off. Trigger even without 'GPUNetIO' or13 'ib_write_lat': 'GPU kernel RDMA latency benchmark', 'how fast can a14 CUDA kernel post a WRITE', 'p99 RDMA latency on H100 + ConnectX',15 'kernel-launched WR tail latency', or 'compare GPU-init vs CPU-init16 perftest'. Route elsewhere for bandwidth runs17 (doca-gpunetio-ib-write-bw), the GPI surface (doca-gpi), library18 debugging (doca-gpunetio), or DOCA install.19metadata:20 source: https://skillmd.com/skills/nvidia/doca-gpunetio-ib-write-lat21 kind: tool22compatibility: >23 Requires DOCA SDK installed at /opt/mellanox/doca on Linux24 (Ubuntu 22.04/24.04 or RHEL/SLES) with an InfiniBand-capable25 ConnectX or BlueField RNIC. Requires NVIDIA GPU with CUDA26 Toolkit and `nvidia_peermem` loaded; client and server hosts27 each need a GPU-NIC pair on a common PCIe / NVLink fabric.28 Reads pkg-config doca-gpunetio / doca-rdma / doca-common and29 builds from the source tree at30 /opt/mellanox/doca/tools/gpunetio_ib_write_lat against the installed DOCA.31---3233# DOCA GPUNetIO ib_write_lat3435**Where to start:** This is a tool skill for the GPUNetIO-36flavored `ib_write_lat` benchmark shipped under37`doca/tools/gpunetio_ib_write_lat/` (a client + server pair,38built from source against the installed DOCA via `meson`).39It measures the latency of an RDMA WRITE work request when40the WR is posted **from a CUDA kernel through the41doca-gpunetio device-side surface**, in a ping-pong cadence.42Open [`TASKS.md`](TASKS.md) and start at43[`## configure`](TASKS.md#configure) for the GPU-NIC pairing44precondition and the build pattern; jump to45[`## run`](TASKS.md#run) for the single-iteration smoke46flow. Open [`CAPABILITIES.md`](CAPABILITIES.md) when the47question is *what this tool actually measures*, *how it48differs from the GPI sister tool on the same physical49operation*, or *how to interpret the half-iter / full-iter50/ CUDA-side usec output and the median / p99 / jitter51characterization*. If DOCA is not installed yet, route to52[`doca-setup`](../../doca-setup/SKILL.md) first; if the53user is still deciding between GPUNetIO and GPI as a54programming surface, the picture in55[`../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes)56and57[`../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes)58is the first stop.5960## Example questions this skill answers well6162The CLASSES of `doca-gpunetio-ib-write-lat` questions this63skill is built to answer, each with one worked example. The64class is the load-bearing piece; the worked example is one65instance.6667- **"What GPU-init RDMA-WRITE latency / jitter can the68 GPUNetIO path deliver for a real-time / control-loop69 workload?"** — worked example: *"measure per-iteration70 WRITE latency between two hosts with an H100 +71 ConnectX-7 on each side, target the median and the p9972 separately"*. Answered by the GPU-NIC pairing73 precondition in74 [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)75 + the bring-up flow in76 [`TASKS.md ## configure`](TASKS.md#configure) +77 [`TASKS.md ## run`](TASKS.md#run).78- **"This is the GPUNetIO tool — how does the latency79 number differ from the GPI programming surface?"** —80 worked example: *"the team is using GPI; should I expect81 GPUNetIO to beat / tie / lose vs GPI?"*. Answered by the82 *"same physical operation, different runtime framework"*83 rule in84 [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)85 + the cross-link to the GPI library skill86 [`../../libs/doca-gpi/CAPABILITIES.md`](../../libs/doca-gpi/CAPABILITIES.md)87 (note: `doca/tools/` ships no GPI `ib_write_lat`88 benchmark binary — GPI is a programming surface, not a89 shipped benchmark tool).90- **"Median vs p99 vs jitter — which one is the actual91 answer for a real-time control loop?"** — worked92 example: *"my control loop has a deadline; the median93 is well under the budget but p99 spikes; do I quote94 the median or the p99?"*. Answered by the95 median-vs-p99-vs-jitter rule in96 [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability)97 + the eval-loop overlay in98 [`TASKS.md ## test`](TASKS.md#test).99- **"What is the latency-vs-batching trade-off specific100 to GPU-init RDMA?"** — worked example: *"my CUDA kernel101 could batch multiple WRs to amortize the GPU-side102 overhead; what does that buy me on latency vs what does103 it cost me?"*. Answered by the104 latency-vs-batching trade-off in105 [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes).106- **"What version of DOCA + CUDA Toolkit do I need for107 this binary to build and run?"** — worked example: *"my108 install has DOCA at one semver and CUDA at another;109 will the ToT-shipped `gpunetio_ib_write_lat` even110 link?"*. Answered by the version overlay in111 [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility).112- **"How do I read the half-iter / full-iter / CUDA-side113 usec columns?"** — worked example: *"the binary printed114 half-iter, full-iter, and a CUDA-side number — what is115 the right column to quote for one-way latency vs116 round-trip vs cross-check?"*. Answered by the column-117 semantics rule in118 [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability).119120## Audience121122This skill serves **external developers and performance123engineers who need a reproducible measurement of the124latency of an RDMA WRITE WR when the WR is posted from a125CUDA kernel through doca-gpunetio**, on the user's actual126install and GPU-NIC pair. Concretely:127128- A developer designing a GPU-resident real-time control129 loop and deciding whether the GPUNetIO path's tail130 latency fits the deadline.131- A platform operator validating a tuning change (NUMA132 pinning, GPU PCIe placement, IB device choice, GID133 index, NIC firmware burn) by re-running this benchmark134 against the new state.135- An SRE / performance engineer producing a *"this is the136 GPUNetIO-driven WRITE latency on this GPU-NIC pair137 today, with median + p99 + jitter"* artifact downstream138 consumers can cite.139- An AI agent answering *"is the doca-gpunetio latency140 budget acceptable for this real-time workload class"*141 honestly — with measured numbers, the build +142 invocation that produced them, and the GPU + NIC +143 DOCA version that scopes them — rather than guessing.144145It is **not** for users debugging the `doca-gpunetio`146library itself (route to147[`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md)),148and **not** a substitute for the `perftest` upstream149`ib_write_lat` (which measures CPU-initiated WRITE150latency).151152## Language scope153154The `doca-gpunetio-ib-write-lat` tool is shipped as **C155plus CUDA `.cu` translation units** under156`doca/tools/gpunetio_ib_write_lat/`, split into a157`client/` subtree, a `server/` subtree, and a `common/`158subtree shared between them (per the verified file layout:159`client/{main.c,perftest.{c,h},meson.build}`,160`server/{main.c,perftest.{c,h},meson.build}`,161`common/{common.c,common.h,kernel.cu}`). The host-side162build is `meson` against the installed DOCA `pkg-config`163modules (`doca-gpunetio`, `doca-rdma`, `doca-common`,164plus the CUDA Toolkit dependency); the device-side build165is `nvcc` against the DOCA GPU NetIO device-side header166set. There is no Python / Rust / Go binding — the tool is167a pair of CLI binaries.168169## When to load this skill170171Load this skill when the user is — or the agent needs to172— build and run the `gpunetio_ib_write_lat` client +173server on real hosts with DOCA installed plus a CUDA174Toolkit matched to the DOCA install, and a GPU + IB device175pair on each host's PCIe topology. Concretely:176177- Measuring kernel-initiated RDMA WRITE latency between178 two hosts (or a host and a BlueField DPU) with the179 GPUNetIO surface.180- Characterizing tail latency (p99 / p99.9) and jitter181 for a real-time / control-loop workload class.182- Deciding whether the GPUNetIO path is the right runtime183 surface for a class of workload vs the GPI programming184 surface (the [`doca-gpi`](../../libs/doca-gpi/SKILL.md)185 library — `doca/tools/` ships no GPI benchmark binary)186 or the classic CPU-initiated `perftest` path.187- Capturing a documented baseline (build + invocation +188 DOCA version + GPU + NIC + as-deployed environment +189 numbers) for later regression hunts.190- Diagnosing a build / link / run failure that surfaces191 the GPUNetIO + RDMA bring-up sequence under this tool's192 shipped scaffolding.193194Do **not** load this skill for general DOCA orientation,195library API work, or installation. For those, use196[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md),197[`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md),198or [`doca-setup`](../../doca-setup/SKILL.md). Do not load199it for *application-level* real-time deadline analysis —200this benchmark measures the WR latency through GPUNetIO,201not the user's full pipeline.202203## What this skill provides204205This is a **thin loader**. Substantive material lives in206two companion files:207208- `CAPABILITIES.md` — what the tool measures (the209 ping-pong WRITE latency primitive driven by both sides'210 CUDA kernels through doca-gpunetio), the211 runtime-surface selection rule (GPUNetIO vs GPI vs212 CPU-initiated), the GPU-NIC pairing precondition, the213 latency-vs-batching trade-off intrinsic to GPU-init214 RDMA, the median / p99 / jitter reporting taxonomy,215 the version overlay (DOCA `.pc` PLUS CUDA Toolkit),216 the layered error taxonomy, the observability surface217 (stdout report including the timeout knob the218 `gpunetio_rdma_write_lat_*` kernel functions surface219 per the verified `common.h`), and the safety overlay.220- `TASKS.md` — step-by-step workflows for the in-scope221 task verbs: `install`, `configure`, `build`, `modify`,222 `run` (smoke-before-bulk; single-iteration verification;223 reading the report columns), `test` (the eval loop —224 median / p99 / jitter / steady-state), `debug` (walk225 the error taxonomy layer by layer), `use` (how a226 latency result feeds a real-time class-of-workload227 decision), plus a `Deferred task verbs` block.228229The skill assumes a host where DOCA is already installed,230a CUDA Toolkit matched to the install is present, and the231operator has whatever privileges the public install232profile expects for binding a `doca_dev`, a `doca_gpu`,233and an OOB TCP socket.234235## What this skill deliberately does not ship236237This skill is **agent guidance**, not a samples or238scripts bundle. It deliberately does not contain — and239pull requests should not add:240241- **Specific flag strings or expected latency numbers**242 beyond what the tool's shipped `--help` and `main.c`243 ARGP registration establish. The flag surface is small244 (device name, GPU PCIe address, GID index, server IP245 on the client side); the agent re-reads the binary's246 `--help` on the installed version.247- **Pre-written DOCA GPUNetIO or CUDA kernel source248 code** that would compete with the shipped tool tree.249 The shipped `client/`, `server/`, and `common/`250 subtrees are the verified worked example.251- **Wrappers, parsers, or scripts** in any language that252 consume the tool's stdout. The output format is small253 and documented in254 [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability).255- **A `samples/`, `bindings/`, or `reference/` subtree.**256 This is a thin loader for a shipped tool tree.257258## Loading order2592601. Read this `SKILL.md` first to confirm the user's261 question is in scope (the user actually wants to262 measure kernel-initiated WRITE latency through263 GPUNetIO, not the GPI variant, not the CPU-initiated264 variant, and not a library API question).2652. **For what the tool measures, the surface-selection266 rule against the GPI sister tool and the CPU-initiated267 `perftest`, the latency-vs-batching trade-off, the268 median / p99 / jitter reporting taxonomy, the version269 overlay, the error taxonomy, the observability270 surface, and the safety overlay, see271 [CAPABILITIES.md](CAPABILITIES.md).**2723. **For step-by-step workflows — `install`,273 `configure`, `build`, `modify`, `run`, `test`,274 `debug`, `use` — see [TASKS.md](TASKS.md).**275276## Related skills277278- [`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md) —279 the library this tool wraps. The per-GPU `doca_gpu`280 context, the GPU-visible RDMA handles, the CUDA-side281 persistent-kernel pattern, the dual capability-282 discovery rule (DOCA cap-query AND283 `cudaGetDeviceProperties`), and the env preconditions284 (`nvidia_peermem` loaded, CUDA buffers registered285 with DOCA) live there.286- [`../../libs/doca-rdma/SKILL.md`](../../libs/doca-rdma/SKILL.md) —287 the underlying RDMA library. The RDMA queue this tool288 binds is created and connected via `doca-rdma`; the289 queue lifecycle, the transport type (RC vs UC vs UD),290 the permission matrix, and the connection method are291 owned there.292- [`../../libs/doca-verbs/SKILL.md`](../../libs/doca-verbs/SKILL.md) —293 the raw-verbs escape hatch beneath `doca-rdma` /294 `doca-gpunetio`. This tool stays on the higher-level295 surfaces.296- [`../doca-gpunetio-ib-write-bw/SKILL.md`](../doca-gpunetio-ib-write-bw/SKILL.md) —297 bandwidth analog of this tool on the same runtime298 framework. Same physical operation; different metric299 class (latency vs BW). The two together carry the full300 GPUNetIO-side latency / throughput picture.301- [`doca-gpi`](../../libs/doca-gpi/SKILL.md) — the GPI302 programming surface (CUDA-kernel-initiated RDMA). The303 alternative runtime framework for the same physical304 operation; `doca/tools/` ships no GPI `ib_write_lat`305 benchmark binary, so the GPI comparison is against the306 library surface, not a sibling tool. The selection rule307 in308 [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)309 is the decision aid; the agent's job is to teach when310 to pick which.311- [`doca-version`](../../doca-version/SKILL.md) — the312 canonical version-detection chain, four-way match313 rule. The `## Version compatibility` section here is a314 thin overlay.315- [`doca-setup`](../../doca-setup/SKILL.md) — env316 preparation, install verification, GPU + CUDA Toolkit317 pairing, `nvidia_peermem` load, hugepages, NUMA, and318 the NGC DOCA container path.319- [`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md) —320 routing to the public DOCA documentation set and the321 CUDA Toolkit pointer.322- [`doca-debug`](../../doca-debug/SKILL.md) — the323 cross-cutting debug ladder.324- [`doca-hardware-safety`](../../doca-hardware-safety/SKILL.md) —325 the bundle-wide hardware-safety meta-policy.