DOCA DPA High-Level Tracer
Where to start: This is a tool skill for invoking
doca_dpa_hl_tracer — the documented host-side CLI that
captures DPA-side execution traces in higher-level terms
(DPA programming events: kernel entry / exit, sync points,
comm primitive calls, RDMA WR submission, completions) rather
than raw cycle counts. Open TASKS.md and start at
## configure for the
mode-vs-overhead decision and the JSON config layout, then
## run for the
capture → decode → render pipeline. Open
CAPABILITIES.md when the question is
what does this tool actually trace, which DPA programming
events does it expose, what is the trace-overhead vs
fidelity tradeoff, or how does it slot into a DPA debug
loop alongside doca-dpa
and doca-debug. If DPA is not
the right surface for the user's question (e.g. the bug is
host-side, the bug is in the DPACC-produced image, the user
wants raw cycle counts), the path-selection rule in
CAPABILITIES.md ## Capabilities and modes
routes the agent before any capture is attempted.
Example questions this skill answers well
The CLASSES of doca_dpa_hl_tracer questions this skill is
built to answer, each with one worked example. The class is
the load-bearing piece; the worked example is one instance.
- "My DPA kernel is doing the wrong thing — where do I
look?" — worked example: "my host-side
doca_dpa_kernel_launch_update_* completes, but the
kernel's reported result is wrong; no host-side
DOCA_ERROR_*". Answered by the when DPA-side
high-level tracing is the right surface gate in
CAPABILITIES.md ## Capabilities and modes
- the
capture → decode → render flow in
TASKS.md ## run + the
which DPA programming events to focus on rule in
TASKS.md ## debug.
- "My DPA kernel is slow at a granularity that doesn't
show up in cycle profiles — how do I see kernel-entry to
first-comm-call latency?" — worked example: "my DPA
kernel runs but the time between launch and the first
RDMA WR submission is bigger than I expected". Answered
by the event-taxonomy table in
CAPABILITIES.md ## Capabilities and modes
- the iterative loop in
TASKS.md ## test which treats trace
overhead, mode (TRACE vs CRIT), and capture window
as axes to tune.
- "How do I capture a trace without burying the DPA in
observation overhead?" — worked example: "
TRACE mode
is producing too much data and my measured DPA latency
went up by 2x compared to without the tracer". Answered
by the mode-vs-overhead tradeoff in
CAPABILITIES.md ## Capabilities and modes
- the
CRIT-first guidance in
TASKS.md ## configure (start
with critical-events-only; widen to TRACE only when the
bug demands per-event detail).
- "My trace file got truncated mid-run — how should I
configure the file-size limits?" — worked example:
"binary trace file hit 5 GB and the capture stopped".
Answered by the
log_file_max_size_in_bytes /
bin_file_max_size_in_bytes / file_size_limit_policy
triple in
CAPABILITIES.md ## Capabilities and modes
- the JSON config layout in
TASKS.md ## configure.
- "Is the tracer on my install, and is it paired with the
matching
doca-dpa library and DPACC compiler
version?" — worked example: "is the tracer ABI on my
install compatible with the DPA image my DPACC just
produced?". Answered by the version-overlay in
CAPABILITIES.md ## Version compatibility,
which redirects to the canonical
doca-version chain and
adds the tracer ↔ doca-dpa library ↔ DPACC compiler
match rule.
- "The capture file looks empty / decode failed — is the
install broken, no events fired, or am I tracing the
wrong thing?" — worked example: "
doca_dpa_hl_tracer
ran, wrote a file, but the parser shows zero events".
Answered by the layered error taxonomy in
CAPABILITIES.md ## Error taxonomy
(install / device-binding / DPA-image-instrumented /
capture-window / decode-vs-elf / overhead-saturated /
version / cross-cutting) + the layered walk in
TASKS.md ## debug.
Audience
This skill serves external developers, platform operators,
and AI agents who have already brought up a DPA-side workload
through doca-dpa and now
need higher-level visibility into what the DPA kernel is
actually doing on the wire — DPA programming events
ordering, sync gaps, comm-call latencies, RDMA-WR / completion
timing — without dropping all the way down to raw cycle
counters. Concretely:
- A DPA developer who can launch their kernel cleanly from
the host side but whose kernel's result is wrong or
whose DPA-side performance is below expectation, and
who needs a DPA-side ground truth before triaging.
- A platform operator running a DPA-using workload (RDMA
offload from accelerator, custom CC algorithm via
doca-pcc) and needs to localize a regression to the
DPA side without instrumenting the application.
- An AI agent producing a DPA-side trace report as
evidence for the host-side
doca-dpa TASKS.md ## debug
ladder when the host side reports clean completions but
the DPA-side behaviour is wrong.
It is not for users debugging the tracer binary itself,
not a substitute for the live public DOCA DPA Tools
guide, not the right place for users learning how to
write a DPA kernel (that audience belongs in
doca-dpa plus the public
DOCA DPA / DPACC / DPA-Comms / DPA-Verbs guides), and not
the right place for raw per-instruction cycle profiling
(different surface, different tool — route via
doca-public-knowledge-map ## DOCA tools).
The tracer is shipped as a CLI binary under
/opt/mellanox/doca/tools/, not a library you link against.
The skill uses the same kind: tool three-file shape as
the rest of the bundle so the agent's task-verb contract is
uniform across libraries, services, and tools.
Language scope
doca_dpa_hl_tracer is a C++ host-side CLI. Its inputs are
its JSON config file, the DPA-side ELF (the
doca_dpa_app-class image produced by DPACC), and a running
DPA-side workload that the host-side doca-dpa lifecycle
already started. Its outputs are a binary trace file
(bin_file) and a human-readable log file (log_file).
The skill keeps the workflow guidance language-neutral —
the DPA-side workload it traces can be C compiled by DPACC
or any other DPA translation unit DPACC accepts — and
routes per-language questions to the public DPA / DPACC
guides via
doca-public-knowledge-map.
When to load this skill
Load this skill when the user is — or the agent needs to —
invoke doca_dpa_hl_tracer on a real host with DOCA
installed against a BlueField with a DPA processor visible to
the host, and the host-side
doca-dpa lifecycle has
already brought a DPA workload up at least once. Concretely:
- Capturing a DPA-side trace to localize a DPA kernel's
wrong-result or wrong-ordering behaviour when the
host-side
doca-dpa lifecycle reports clean completions.
- Capturing a DPA-side trace to localize a DPA-side
performance gap (kernel-entry to first-comm latency,
RDMA-WR-issue to completion gap, sync-point dwell time)
at a granularity above raw cycle counts.
- Choosing between
TRACE and CRIT capture modes based
on the bug-vs-overhead tradeoff and the available
capture window.
- Tuning the JSON config (thread priorities, core
affinities, file size limits, file-size-limit policy) so
the capture itself does not perturb the workload more
than the bug it is investigating.
- Decoding a captured
bin_file against the matching
DPA-side ELF to render the human-readable event stream.
- Capturing a side-effect-bounded trace as prerequisite
evidence for a host-side
doca-dpa TASKS.md ## debug
ladder step.
Do not load this skill for general DOCA orientation,
DPA-side programming model questions, raw cycle profiling,
or DOCA / DPACC install. For those, route to
doca-public-knowledge-map,
doca-dpa, or
doca-setup.
What this skill provides
This is a thin loader. Substantive material lives in two
companion files:
CAPABILITIES.md — what doca_dpa_hl_tracer captures: the
DPA programming event taxonomy (kernel entry / exit, sync
points, comm primitive calls, RDMA WR submission and
completion drain), the two documented capture modes
(TRACE for full per-event, CRIT for critical-events
only), the trace-overhead-vs-fidelity tradeoff, the
config-file shape (receiver / binary-writer / file-writer
/ printer threads with priority + core affinity, file
size limits, file_size_limit_policy), the
capture-window + workload-must-be-running invariant, the
ELF-must-match-image rule for decode, the
version-availability overlay (tracer ↔
doca-dpa library ↔
DPACC compiler), the layered error taxonomy
(install / device-binding / image-instrumented /
capture-window / decode / overhead-saturated / version /
cross-cutting), the observability surface (binary trace
file + log file + tool's own stderr), and the safety
policy (capture is bounded; tracing is not a production
observability surface).
TASKS.md — step-by-step workflows for the in-scope
task verbs: install (route to host-side DOCA install +
DPA prerequisites), configure (mode + JSON config
layout + capture window), build (route to install —
the binary is shipped, the DPA-side application is
user-built by DPACC), modify (refuse — do not patch
the binary; modify the JSON config and the invocation
instead), run (the capture flow with --mode,
--config-file, --output-file), test (iterative
loop tuning mode, window, and overhead), debug (walk
the error taxonomy), use (consume the decoded trace
in a doca-dpa debug session), plus a Deferred task verbs block.
The skill assumes a host where DOCA is already installed at
the standard location, a BlueField with a DPA processor is
present and visible to the host, the DPACC compiler is
installed at a version matched to the host-side DOCA, the
DPA-side application image (the ELF the tracer decodes
against) is on disk and matches what the
doca-dpa lifecycle loaded,
and the operator has the privileges the public DOCA DPA
Tools guide requires.
What this skill deliberately does not ship
This skill is agent guidance, not a samples or scripts
bundle. To keep the boundary clean, it deliberately does not
contain — and pull requests should not add:
- Specific flag strings, event names, or mode tokens
beyond what the public DOCA DPA Tools page and
--help
document. The DPA programming events surface evolves
release to release; --help on the installed binary is
the authoritative inventory.
- Pre-baked example traces or expected event timings.
Trace output is workload-, DPA-image-, BlueField-, and
firmware-specific; a captured example pinned to one
setup misleads operators elsewhere.
- Wrappers, parsers, or rendering scripts in any
language that consume the binary trace format. The
format is documented; users who want to script against
it should read the live guide and write the parser
against their installed version.
- A specific tuning recommendation derived from a single
trace. A DPA-side perf decision (move a sync, batch a
comm call, change a launch argument) is a workload
question and the skill prescribes how to capture and
read traces — it refuses to translate a captured gap
into a kernel-rewrite recommendation without the user's
own analysis.
- A
samples/ or reference/ subtree. This is a thin
loader for a shipped CLI; substantive material lives on
the public page, in --help, and in
doca-dpa.
Loading order
- Read this
SKILL.md first to confirm the user's question
is in scope (DPA-side high-level tracing, not DPA-side
programming and not raw cycle profiling).
- For the event taxonomy, capture modes, overhead
tradeoff, JSON config layout, version overlay, error
taxonomy, observability, and safety policy, see
CAPABILITIES.md.
- For the documented invocations and the
capture → decode → render workflow —
install,
configure, build, modify, run, test, debug,
use — see TASKS.md.
Related skills
doca-dpa — the host-side
DPA control library whose loaded application image the
tracer captures. Pair them in every DPA debug session:
doca-dpa brings the workload up; the tracer captures
what the workload does at the DPA programming event
layer. Conflating the library with the tracer is the
most common DPA-debug first-touch error.
doca-debug — the
cross-cutting debug ladder. The tracer slots in at the
runtime layer as the DPA-side ground truth before any
DPA-side perf or correctness conclusion is made.
doca-public-knowledge-map
— routing to the public DOCA DPA Tools page on
docs.nvidia.com and the rest of the public DOCA
documentation set.
doca-version — canonical
DOCA version-handling rules. The ## Version compatibility section in CAPABILITIES.md
is a concise overlay that redirects here for the body
and adds the tracer ↔ doca-dpa library ↔ DPACC
compiler matching rule.
doca-setup — env
preparation, install verification, DPACC compiler
install / verification, BlueField mode (the DPA
processor must be exposed before any tracing is
meaningful), and the I have no install yet path with
the public NGC DOCA container.
doca-structured-tools-contract
— the bundle's detect → prefer → fall back → report
contract for structured helper tools. The command
appendix in TASKS.md honors this contract.
doca-programming-guide
— general DOCA programming patterns shared by every
library / tool surface, including the cross-library
DOCA_ERROR_* taxonomy this tool's host-side error
layer overlays on top of when host-side doca-dpa
calls fail in tandem.
The DPA-side companion libraries doca-dpa-comms (comm
primitives the DPA kernel itself calls) and
doca-dpa-verbs (RDMA verbs the DPA kernel itself calls)
are different artifacts that the tracer's DPA
programming events surface visibly names; for the
DPA-side programming model itself, route through
doca-public-knowledge-map
to the public DOCA DPA-Comms and DPA-Verbs guides and to
the shipped /opt/mellanox/doca/samples/doca_dpa/ samples.
This tool traces their use; it does not redefine them.
1---2license: Apache-2.03name: doca-dpa-hl-tracer4description: >5 Use this skill when the user runs doca_dpa_hl_tracer to6 capture/decode DPA-side traces at the programming-events7 layer (kernel entry/exit, sync points, comm primitive8 calls, RDMA WR submission, completion drain) — picking9 TRACE vs CRIT, tuning the JSON config (file-size limits10 + file_size_limit_policy, thread priorities/cores),11 decoding against the matching DPA-side ELF, or12 diagnosing empty/noisy captures. Trigger even when the13 user does not explicitly mention "DOCA DPA tracer" or14 "high-level tracer" — typical implicit phrasings include15 "DPA kernel returns wrong result but host completions16 look clean", "kernel-entry to first-comm latency is17 huge", "RDMA WR to drain gap on the DPA", "trace file18 truncated mid-run", "TRACE doubled my DPA latency", or19 "tracer wrote a file but parser shows zero events".20 Refuse and route elsewhere for writing DPA kernels,21 DPA-Comms/DPA-Verbs programming, raw per-cycle DPA22 profiling, host-side doca-dpa debugging, or production23 DPA telemetry — those belong to other skills.24metadata:25 source: https://skillmd.com/skills/nvidia/doca-dpa-hl-tracer26 kind: tool27compatibility: >28 Requires DOCA SDK installed at /opt/mellanox/doca on29 Linux (Ubuntu 22.04/24.04 or RHEL/SLES) with a BlueField30 device whose DPA processor is exposed to the host, plus31 the DOCA DPA Tools optional component (binary at32 /opt/mellanox/doca/tools/doca_dpa_hl_tracer). Requires a33 DPACC-built DPA-side ELF and a live doca-dpa-launched34 workload for events to fire.35---3637# DOCA DPA High-Level Tracer3839**Where to start:** This is a tool skill for invoking40`doca_dpa_hl_tracer` — the documented host-side CLI that41captures DPA-side execution traces in higher-level terms42(DPA programming events: kernel entry / exit, sync points,43comm primitive calls, RDMA WR submission, completions) rather44than raw cycle counts. Open [`TASKS.md`](TASKS.md) and start at45[`## configure`](TASKS.md#configure) for the46mode-vs-overhead decision and the JSON config layout, then47[`## run`](TASKS.md#run) for the48capture → decode → render pipeline. Open49[`CAPABILITIES.md`](CAPABILITIES.md) when the question is50*what does this tool actually trace*, *which DPA programming51events does it expose*, *what is the trace-overhead vs52fidelity tradeoff*, or *how does it slot into a DPA debug53loop alongside [`doca-dpa`](../../libs/doca-dpa/SKILL.md)54and [`doca-debug`](../../doca-debug/SKILL.md)*. If DPA is not55the right surface for the user's question (e.g. the bug is56host-side, the bug is in the DPACC-produced image, the user57wants raw cycle counts), the path-selection rule in58[`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)59routes the agent before any capture is attempted.6061## Example questions this skill answers well6263The CLASSES of `doca_dpa_hl_tracer` questions this skill is64built to answer, each with one worked example. The class is65the load-bearing piece; the worked example is one instance.6667- **"My DPA kernel is doing the wrong thing — where do I68 look?"** — worked example: *"my host-side69 `doca_dpa_kernel_launch_update_*` completes, but the70 kernel's reported result is wrong; no host-side71 `DOCA_ERROR_*`"*. Answered by the *when DPA-side72 high-level tracing is the right surface* gate in73 [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)74 + the75 capture → decode → render flow in76 [`TASKS.md ## run`](TASKS.md#run) + the77 *which DPA programming events to focus on* rule in78 [`TASKS.md ## debug`](TASKS.md#debug).79- **"My DPA kernel is slow at a granularity that doesn't80 show up in cycle profiles — how do I see kernel-entry to81 first-comm-call latency?"** — worked example: *"my DPA82 kernel runs but the time between launch and the first83 RDMA WR submission is bigger than I expected"*. Answered84 by the event-taxonomy table in85 [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)86 + the iterative loop in87 [`TASKS.md ## test`](TASKS.md#test) which treats trace88 overhead, mode (`TRACE` vs `CRIT`), and capture window89 as axes to tune.90- **"How do I capture a trace without burying the DPA in91 observation overhead?"** — worked example: *"`TRACE` mode92 is producing too much data and my measured DPA latency93 went up by 2x compared to without the tracer"*. Answered94 by the mode-vs-overhead tradeoff in95 [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)96 + the `CRIT`-first guidance in97 [`TASKS.md ## configure`](TASKS.md#configure) (start98 with critical-events-only; widen to `TRACE` only when the99 bug demands per-event detail).100- **"My trace file got truncated mid-run — how should I101 configure the file-size limits?"** — worked example:102 *"binary trace file hit 5 GB and the capture stopped"*.103 Answered by the `log_file_max_size_in_bytes` /104 `bin_file_max_size_in_bytes` / `file_size_limit_policy`105 triple in106 [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)107 + the JSON config layout in108 [`TASKS.md ## configure`](TASKS.md#configure).109- **"Is the tracer on my install, and is it paired with the110 matching `doca-dpa` library and DPACC compiler111 version?"** — worked example: *"is the tracer ABI on my112 install compatible with the DPA image my DPACC just113 produced?"*. Answered by the version-overlay in114 [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility),115 which redirects to the canonical116 [`doca-version`](../../doca-version/SKILL.md) chain and117 adds the *tracer ↔ `doca-dpa` library ↔ DPACC compiler*118 match rule.119- **"The capture file looks empty / decode failed — is the120 install broken, no events fired, or am I tracing the121 wrong thing?"** — worked example: *"`doca_dpa_hl_tracer`122 ran, wrote a file, but the parser shows zero events"*.123 Answered by the layered error taxonomy in124 [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)125 (install / device-binding / DPA-image-instrumented /126 capture-window / decode-vs-elf / overhead-saturated /127 version / cross-cutting) + the layered walk in128 [`TASKS.md ## debug`](TASKS.md#debug).129130## Audience131132This skill serves **external developers, platform operators,133and AI agents who have already brought up a DPA-side workload134through [`doca-dpa`](../../libs/doca-dpa/SKILL.md) and now135need higher-level visibility into what the DPA kernel is136actually doing on the wire** — DPA programming events137ordering, sync gaps, comm-call latencies, RDMA-WR / completion138timing — without dropping all the way down to raw cycle139counters. Concretely:140141- A DPA developer who can launch their kernel cleanly from142 the host side but whose kernel's *result* is wrong or143 whose *DPA-side performance* is below expectation, and144 who needs a DPA-side ground truth before triaging.145- A platform operator running a DPA-using workload (RDMA146 offload from accelerator, custom CC algorithm via147 `doca-pcc`) and needs to localize a regression to the148 DPA side without instrumenting the application.149- An AI agent producing a *DPA-side trace report* as150 evidence for the host-side151 [`doca-dpa TASKS.md ## debug`](../../libs/doca-dpa/TASKS.md#debug)152 ladder when the host side reports clean completions but153 the DPA-side behaviour is wrong.154155It is **not** for users debugging the tracer binary itself,156**not** a substitute for the live public DOCA DPA Tools157guide, **not** the right place for users learning how to158write a DPA kernel (that audience belongs in159[`doca-dpa`](../../libs/doca-dpa/SKILL.md) plus the public160DOCA DPA / DPACC / DPA-Comms / DPA-Verbs guides), and **not**161the right place for raw per-instruction cycle profiling162(different surface, different tool — route via163[`doca-public-knowledge-map ## DOCA tools`](../../doca-public-knowledge-map/SKILL.md#doca-tools)).164165The tracer is shipped as a **CLI binary** under166`/opt/mellanox/doca/tools/`, not a library you link against.167The skill uses the same `kind: tool` three-file shape as168the rest of the bundle so the agent's task-verb contract is169uniform across libraries, services, and tools.170171## Language scope172173`doca_dpa_hl_tracer` is a C++ host-side CLI. Its inputs are174its JSON config file, the DPA-side ELF (the175`doca_dpa_app`-class image produced by DPACC), and a running176DPA-side workload that the host-side `doca-dpa` lifecycle177already started. Its outputs are a binary trace file178(`bin_file`) and a human-readable log file (`log_file`).179The skill keeps the workflow guidance language-neutral —180the DPA-side workload it traces can be C compiled by DPACC181or any other DPA translation unit DPACC accepts — and182routes per-language questions to the public DPA / DPACC183guides via184[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md).185186## When to load this skill187188Load this skill when the user is — or the agent needs to —189invoke `doca_dpa_hl_tracer` on a real host with DOCA190installed against a BlueField with a DPA processor visible to191the host, and the host-side192[`doca-dpa`](../../libs/doca-dpa/SKILL.md) lifecycle has193already brought a DPA workload up at least once. Concretely:194195- Capturing a DPA-side trace to localize a DPA kernel's196 wrong-result or wrong-ordering behaviour when the197 host-side `doca-dpa` lifecycle reports clean completions.198- Capturing a DPA-side trace to localize a DPA-side199 performance gap (kernel-entry to first-comm latency,200 RDMA-WR-issue to completion gap, sync-point dwell time)201 at a granularity above raw cycle counts.202- Choosing between `TRACE` and `CRIT` capture modes based203 on the bug-vs-overhead tradeoff and the available204 capture window.205- Tuning the JSON config (thread priorities, core206 affinities, file size limits, file-size-limit policy) so207 the capture itself does not perturb the workload more208 than the bug it is investigating.209- Decoding a captured `bin_file` against the matching210 DPA-side ELF to render the human-readable event stream.211- Capturing a side-effect-bounded trace as prerequisite212 evidence for a host-side213 [`doca-dpa TASKS.md ## debug`](../../libs/doca-dpa/TASKS.md#debug)214 ladder step.215216Do **not** load this skill for general DOCA orientation,217DPA-side programming model questions, raw cycle profiling,218or DOCA / DPACC install. For those, route to219[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md),220[`doca-dpa`](../../libs/doca-dpa/SKILL.md), or221[`doca-setup`](../../doca-setup/SKILL.md).222223## What this skill provides224225This is a **thin loader**. Substantive material lives in two226companion files:227228- `CAPABILITIES.md` — what `doca_dpa_hl_tracer` captures: the229 DPA programming event taxonomy (kernel entry / exit, sync230 points, comm primitive calls, RDMA WR submission and231 completion drain), the two documented capture modes232 (`TRACE` for full per-event, `CRIT` for critical-events233 only), the trace-overhead-vs-fidelity tradeoff, the234 config-file shape (receiver / binary-writer / file-writer235 / printer threads with priority + core affinity, file236 size limits, `file_size_limit_policy`), the237 capture-window + workload-must-be-running invariant, the238 ELF-must-match-image rule for decode, the239 version-availability overlay (tracer ↔240 [`doca-dpa`](../../libs/doca-dpa/SKILL.md) library ↔241 DPACC compiler), the layered error taxonomy242 (install / device-binding / image-instrumented /243 capture-window / decode / overhead-saturated / version /244 cross-cutting), the observability surface (binary trace245 file + log file + tool's own stderr), and the safety246 policy (capture is bounded; tracing is not a production247 observability surface).248- `TASKS.md` — step-by-step workflows for the in-scope249 task verbs: `install` (route to host-side DOCA install +250 DPA prerequisites), `configure` (mode + JSON config251 layout + capture window), `build` (route to install —252 the binary is shipped, the DPA-side application is253 user-built by DPACC), `modify` (refuse — do not patch254 the binary; modify the JSON config and the invocation255 instead), `run` (the capture flow with `--mode`,256 `--config-file`, `--output-file`), `test` (iterative257 loop tuning mode, window, and overhead), `debug` (walk258 the error taxonomy), `use` (consume the decoded trace259 in a `doca-dpa` debug session), plus a `Deferred task260 verbs` block.261262The skill assumes a host where DOCA is already installed at263the standard location, a BlueField with a DPA processor is264present and visible to the host, the DPACC compiler is265installed at a version matched to the host-side DOCA, the266DPA-side application image (the ELF the tracer decodes267against) is on disk and matches what the268[`doca-dpa`](../../libs/doca-dpa/SKILL.md) lifecycle loaded,269and the operator has the privileges the public DOCA DPA270Tools guide requires.271272## What this skill deliberately does not ship273274This skill is **agent guidance**, not a samples or scripts275bundle. To keep the boundary clean, it deliberately does not276contain — and pull requests should not add:277278- **Specific flag strings, event names, or mode tokens279 beyond what the public DOCA DPA Tools page and `--help`280 document.** The DPA programming events surface evolves281 release to release; `--help` on the installed binary is282 the authoritative inventory.283- **Pre-baked example traces or expected event timings.**284 Trace output is workload-, DPA-image-, BlueField-, and285 firmware-specific; a captured example pinned to one286 setup misleads operators elsewhere.287- **Wrappers, parsers, or rendering scripts** in any288 language that consume the binary trace format. The289 format is documented; users who want to script against290 it should read the live guide and write the parser291 against their installed version.292- **A specific tuning recommendation derived from a single293 trace.** A DPA-side perf decision (move a sync, batch a294 comm call, change a launch argument) is a workload295 question and the skill prescribes how to *capture and296 read* traces — it refuses to translate a captured gap297 into a kernel-rewrite recommendation without the user's298 own analysis.299- **A `samples/` or `reference/` subtree.** This is a thin300 loader for a shipped CLI; substantive material lives on301 the public page, in `--help`, and in302 [`doca-dpa`](../../libs/doca-dpa/SKILL.md).303304## Loading order3053061. Read this `SKILL.md` first to confirm the user's question307 is in scope (DPA-side high-level tracing, not DPA-side308 programming and not raw cycle profiling).3092. **For the event taxonomy, capture modes, overhead310 tradeoff, JSON config layout, version overlay, error311 taxonomy, observability, and safety policy, see312 [CAPABILITIES.md](CAPABILITIES.md).**3133. **For the documented invocations and the314 capture → decode → render workflow — `install`,315 `configure`, `build`, `modify`, `run`, `test`, `debug`,316 `use` — see [TASKS.md](TASKS.md).**317318## Related skills319320- [`doca-dpa`](../../libs/doca-dpa/SKILL.md) — the host-side321 DPA control library whose loaded application image the322 tracer captures. Pair them in every DPA debug session:323 `doca-dpa` brings the workload up; the tracer captures324 what the workload does at the DPA programming event325 layer. Conflating the library with the tracer is the326 most common DPA-debug first-touch error.327- [`doca-debug`](../../doca-debug/SKILL.md) — the328 cross-cutting debug ladder. The tracer slots in at the329 *runtime* layer as the DPA-side ground truth before any330 DPA-side perf or correctness conclusion is made.331- [`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md)332 — routing to the public DOCA DPA Tools page on333 `docs.nvidia.com` and the rest of the public DOCA334 documentation set.335- [`doca-version`](../../doca-version/SKILL.md) — canonical336 DOCA version-handling rules. The `## Version337 compatibility` section in [`CAPABILITIES.md`](CAPABILITIES.md)338 is a concise overlay that redirects here for the body339 and adds the *tracer ↔ `doca-dpa` library ↔ DPACC340 compiler* matching rule.341- [`doca-setup`](../../doca-setup/SKILL.md) — env342 preparation, install verification, DPACC compiler343 install / verification, BlueField mode (the DPA344 processor must be exposed before any tracing is345 meaningful), and the *I have no install yet* path with346 the public NGC DOCA container.347- [`doca-structured-tools-contract`](../../doca-structured-tools-contract/SKILL.md)348 — the bundle's detect → prefer → fall back → report349 contract for structured helper tools. The command350 appendix in [`TASKS.md`](TASKS.md) honors this contract.351- [`doca-programming-guide`](../../doca-programming-guide/SKILL.md)352 — general DOCA programming patterns shared by every353 library / tool surface, including the cross-library354 `DOCA_ERROR_*` taxonomy this tool's host-side error355 layer overlays on top of when host-side `doca-dpa`356 calls fail in tandem.357358The DPA-side companion libraries `doca-dpa-comms` (comm359primitives the DPA kernel itself calls) and360`doca-dpa-verbs` (RDMA verbs the DPA kernel itself calls)361are **different artifacts** that the tracer's *DPA362programming events* surface visibly names; for the363DPA-side programming model itself, route through364[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md)365to the public DOCA DPA-Comms and DPA-Verbs guides and to366the shipped `/opt/mellanox/doca/samples/doca_dpa/` samples.367This tool *traces* their use; it does not redefine them.