Edge-LLM operation and plugin extension
An operation is one contract consumed by up to five places: the PyTorch
stub, the ONNX translation, the direct-builder lowering, the C++ plugin, and
the runtime that owns its buffers and weights. Read the canonical documents
first; this skill adds the cross-file wiring and the evidence gates.
docs/source/developer_guide/customization/tensorrt-plugins.md: tensor
contracts and kernel sources for four plugins only; cpp/plugins/ is the full set.
docs/source/developer_guide/customization/customization-guide.md, section
"Custom Operators": the ONNX export extension points.
docs/source/developer_guide/software-design/onnxless-builder.md, sections
"Operation API And TensorRT Lowering" and "Adding An Operation".
AGENTS.md (CUDA / GPU anti-patterns) and CODING_GUIDELINES.md.
Before choosing the wiring, read one recent merged plugin addition as a template
(git log --oneline --no-merges -n 10 origin/main -- cpp/plugins) and note where
this operation differs. Do not use a pre-PluginV3 implementation as a template.
Instructions
Write the operation contract first, choose the smallest surface, wire both routes where required, implement the plugin, test, and gather end-to-end evidence in steps 1 to 5. Report with Completion evidence.
1. Write the contract, then choose
Before code, write down for every input and output: name, rank, which dims are
dynamic, dtype, format, accumulation precision, supported SMs and platforms,
and who owns the memory (engine binding, per-enqueue workspace, runtime pool,
or prepared weight). Then choose:
| Choose |
When |
Reason |
| Native TensorRT layers |
Standard ONNX ops, trt-domain ops, or Net methods express it within the accuracy and latency budget on every claimed platform |
No plugin ABI to version; TensorRT can fuse it; both routes get it from one lowering |
| Extend an existing plugin |
An existing creator already owns the semantics; only an attribute, dtype, or dispatch case is missing |
A second creator for the same semantics duplicates the whole contract |
| Edge-LLM IPluginV3 plugin |
Needs a custom kernel, a packed weight layout, persistent state addressed by explicit indices, or a buffer layout native layers do not preserve (QkvConcatPlugin) |
Only a plugin fixes dtype, layout, and kernel choice across both builders |
| Runtime-only |
Work between engine executions or on host-scheduled state: sampling, embedding and PLE cache I/O, KV and state bookkeeping, weight preparation |
Keeps it out of the engine; lives in cpp/runtime/ with kernels in cpp/kernels/ |
Refuse a plugin that re-implements a native layer without a measured reason,
and refuse a contract change hidden inside an unrelated change.
2. Wire both routes
The file-by-file map for both routes, the worked example, and the weight-layout
alignment points are in references/wiring-map.md. Read it before editing.
Invariants that are not obvious from any single file:
- ONNX node
op_type equals the creator's getPluginName(); the domain is
trt_edgellm; attribute names equal creator field names; plugin_version,
when emitted, equals getPluginVersion().
- The direct route resolves creators through
_OPERATION_CREATORS in
tensorrt_edgellm/builder/ops/backend.py, always with version "1" and an empty
namespace. A version bump breaks that route unless Net._creator changes too.
Net._operation_field encodes only INT32 and FLOAT32 fields. Keep creator
fields to those types so both routes encode them identically.
torch.export drops keyword arguments that equal their defaults, so an
optional attribute needs the same default in the stub, the translation, and
the creator.
- When a change reaches both frontends, update both. Model-specific code stays
in that family's folder under
tensorrt_edgellm/models/ and
tensorrt_edgellm/builder/models/.
3. Implement the C++ plugin
cpp/CMakeLists.txt globs plugins/*.cpp into NvInfer_edgellm_plugin and
recursively globs kernels/*.cpp plus kernels/*.cu into edgellmKernels
(PLUGIN_JIT_* sources go to edgellmPluginJit instead). A .cu file under
cpp/plugins/ is silently not built: put CUDA code in cpp/kernels/. Kernel
artifacts and CuTe DSL groups belong to edge-llm-cutedsl-kernel.
- Derive from
IPluginV3, IPluginV3OneCore, IPluginV3OneBuild, and
IPluginV3OneRuntime; return each from getCapabilityInterface. Register
an IPluginCreatorV3One with REGISTER_TENSORRT_PLUGIN in namespace
trt_edgellm::plugins, which is the C++ namespace only: the TensorRT plugin
namespace stays empty (""). Keep name and version in named constants.
- Both builders create strongly typed networks (
cpp/builder/builderUtils.cpp,
tensorrt_edgellm/builder/core/builder.py). getOutputDataTypes must derive
every output dtype from input dtypes and attributes, and
supportsFormatCombination must accept exactly the contract.
- Build output shapes with
IExprBuilder; require constant dims the kernel
depends on. Validate min, opt, and max in configurePlugin, and revalidate
in onShapeChange and enqueue. On violation, log which tensor, dtype, or
shape is wrong and return -1; never throw across the library boundary.
getFieldsToSerialize writes every attribute plus any build-derived state
the runtime needs. The creator accepts that set in the runtime phase and
returns nullptr for missing or unknown fields, or, with a log naming the
missing group, when an optional kernel group is not compiled in.
- Per-enqueue scratch goes through
getWorkspaceSize. Persistent state is an
engine binding or a runtime-owned pool. Weights are bindings prepared by the
runtime. Do not allocate memory or repack weights in enqueue.
- Launch only on the
stream passed to enqueue; no default stream and no
synchronous CUDA API. Check every CUDA return value.
- Reuse
cpp/plugins/utils/pluginUtils.h (parsePluginScalarField,
Serializer, kDEVICE_ALIGNMENT) and rt::Tensor instead of adding helpers
or raw owning pointers.
- An output that aliases an input also needs builder gating, as
usesAliasedPluginIO in cpp/builder/llmBuilder.cpp does for
DFlashTargetKVCacheUpdatePlugin (plugin name DFlashTargetKVCacheUpdate).
C++ builders and the example apps load the library through
loadEdgellmPluginLib() in cpp/common/trtUtils.h. Without --plugin-path, the direct builder uses the
installed package's plugin and only then build/lib/libNvInfer_edgellm_plugin.so,
so a stale installed library shadows a source build: pass --plugin-path.
4. Tests
Extend the nearest existing test; do not add a harness.
| Contract |
Extend |
| Stub semantics, emitted node, attributes, schema |
the pattern in tests/python-unittests/test_ple_pool_export.py (not itself in the L0 list): eager output vs reference, onnx.checker with check_custom_domain=True |
| Creator, shape and type negotiation, execution, rejection |
PluginRunner in tests/python-unittests/test_plugin_base.py, as test_qkv_concat_plugin.py uses it; build serializes and deserializes the engine, so execution also covers the runtime-phase creator; assert rejection with expect_unsupported=True |
| Direct lowering: name, input order, attributes |
Fake-Net recording tests, patterned on tests/python-unittests/test_experimental_attention_ragged_contract.py (not in the L0 list) |
| Plugin or kernel internals in C++ |
unittests/cpp/plugins/<plugin>/ or unittests/cpp/kernels/<dir>/, registered in an edgellm_add_unit_test call in unittests/CMakeLists.txt (configure fails on unregistered sources); follow edge-llm-cpp-test |
CI runs only tests named in tests/test_lists/l0_python_ut.yml, so list an added
Python test there; a GPU plugin test must also match *plugin*.py to run in the
device job. Cover every
dtype and rank the contract claims and at least one rejection path that
returns an actionable message. Mutation-test each added assertion as
edge-llm-cpp-test and edge-llm-e2e-test describe.
5. End-to-end evidence
For every claimed route, platform, and precision: export or direct-build a
representative model that enqueues the operation, build the engine, run
inference, and compare deterministically against the PyTorch reference.
Use edge-llm-convert-model to build and run it, edge-llm-e2e-test for the
model case, and edge-llm-few-layer-test when the change adds a layer, kernel,
or quantization recipe. Route accuracy failures to
edge-llm-accuracy-debugging.
Not evidence: an ONNX file without an engine; a PluginRunner pass alone; a
collection-only run; a skipped platform; an engine built from a stale plugin
library.
Anti-patterns
| Do not |
Reason |
| Add, remove, or reorder plugin inputs when an attribute or existing input suffices |
Every input change breaks both translations, the bindings, and all serialized engines |
| Reinterpret a serialized field or input without rebuilding engines |
Old engines load and silently compute wrong values |
| Select output dtype from builder precision flags |
Networks are strongly typed; the plugin owns its dtypes |
Allocate, repack, or synchronize in enqueue |
Hot-path rule in AGENTS.md; breaks CUDA graph capture and latency |
Put creator names or PluginField construction in model files |
Lowering is private to tensorrt_edgellm/builder/ops/backend.py and the translation table |
| Fall back silently when an optional kernel group is missing |
The creator must fail with the missing group named; any fallback is explicit and logged |
Completion evidence
Report the revision; the decision row from section 1 with its reason; files changed per layer; focused test commands with
pass, fail, and skip counts; and, for each route, platform, and precision, the
build and inference commands, GPU and SM, engine artifacts, reference
comparison, and verdict. State plainly what was not run and why.
Examples
- "Add a fused operation used by the ONNX and checkpoint-direct builders": write the contract, wire both routes, implement the C++ plugin, then run unit and end-to-end tests.
- "Change an existing plugin's tensor layout": update the contract, every consumer, and the tests, then rebuild engines, because old engines load and silently compute wrong values.
1---2name: edge-llm-plugin-extension3license: Apache-2.04metadata:5 source: https://skillmd.com/skills/nvidia/edge-llm-plugin-extension6 author: TensorRT Edge-LLM Team <noreply@nvidia.com>7 tags: [edge-llm, plugin, tensorrt, onnx, custom-op]8description: Add or change a TensorRT Edge-LLM operation (TensorRT IPluginV3 plugin, native TensorRT layer composition, or runtime-only operation) so one tensor, dtype, shape, layout, serialization, and weight-layout contract holds across the ONNX export route, the experimental checkpoint-direct builder, the C++ engine builders, libNvInfer_edgellm_plugin, and the runtime. Use when adding a custom op, plugin creator field, plugin input or output, dtype, weight repacking, or external-weight recipe; when a plugin is not found or rejects a configuration at engine build; or when choosing between a plugin and native TensorRT layers. Do not use for writing CuTe DSL kernels (use edge-llm-cutedsl-kernel), C++ test discipline (edge-llm-cpp-test), model CI cases (edge-llm-e2e-test), or diagnosing wrong model output (edge-llm-few-layer-test, then edge-llm-accuracy-debugging).9---1011<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->12<!-- SPDX-License-Identifier: Apache-2.0 -->1314# Edge-LLM operation and plugin extension1516An operation is one contract consumed by up to five places: the PyTorch17stub, the ONNX translation, the direct-builder lowering, the C++ plugin, and18the runtime that owns its buffers and weights. Read the canonical documents19first; this skill adds the cross-file wiring and the evidence gates.2021- `docs/source/developer_guide/customization/tensorrt-plugins.md`: tensor22 contracts and kernel sources for four plugins only; `cpp/plugins/` is the full set.23- `docs/source/developer_guide/customization/customization-guide.md`, section24 "Custom Operators": the ONNX export extension points.25- `docs/source/developer_guide/software-design/onnxless-builder.md`, sections26 "Operation API And TensorRT Lowering" and "Adding An Operation".27- `AGENTS.md` (CUDA / GPU anti-patterns) and `CODING_GUIDELINES.md`.2829Before choosing the wiring, read one recent merged plugin addition as a template30(`git log --oneline --no-merges -n 10 origin/main -- cpp/plugins`) and note where31this operation differs. Do not use a pre-PluginV3 implementation as a template.3233## Instructions3435Write the operation contract first, choose the smallest surface, wire both routes where required, implement the plugin, test, and gather end-to-end evidence in steps 1 to 5. Report with Completion evidence.3637## 1. Write the contract, then choose3839Before code, write down for every input and output: name, rank, which dims are40dynamic, dtype, format, accumulation precision, supported SMs and platforms,41and who owns the memory (engine binding, per-enqueue workspace, runtime pool,42or prepared weight). Then choose:4344| Choose | When | Reason |45|---|---|---|46| Native TensorRT layers | Standard ONNX ops, `trt`-domain ops, or `Net` methods express it within the accuracy and latency budget on every claimed platform | No plugin ABI to version; TensorRT can fuse it; both routes get it from one lowering |47| Extend an existing plugin | An existing creator already owns the semantics; only an attribute, dtype, or dispatch case is missing | A second creator for the same semantics duplicates the whole contract |48| Edge-LLM IPluginV3 plugin | Needs a custom kernel, a packed weight layout, persistent state addressed by explicit indices, or a buffer layout native layers do not preserve (`QkvConcatPlugin`) | Only a plugin fixes dtype, layout, and kernel choice across both builders |49| Runtime-only | Work between engine executions or on host-scheduled state: sampling, embedding and PLE cache I/O, KV and state bookkeeping, weight preparation | Keeps it out of the engine; lives in `cpp/runtime/` with kernels in `cpp/kernels/` |5051Refuse a plugin that re-implements a native layer without a measured reason,52and refuse a contract change hidden inside an unrelated change.5354## 2. Wire both routes5556The file-by-file map for both routes, the worked example, and the weight-layout57alignment points are in `references/wiring-map.md`. Read it before editing.58Invariants that are not obvious from any single file:5960- ONNX node `op_type` equals the creator's `getPluginName()`; the domain is61 `trt_edgellm`; attribute names equal creator field names; `plugin_version`,62 when emitted, equals `getPluginVersion()`.63- The direct route resolves creators through `_OPERATION_CREATORS` in64 `tensorrt_edgellm/builder/ops/backend.py`, always with version `"1"` and an empty65 namespace. A version bump breaks that route unless `Net._creator` changes too.66- `Net._operation_field` encodes only INT32 and FLOAT32 fields. Keep creator67 fields to those types so both routes encode them identically.68- `torch.export` drops keyword arguments that equal their defaults, so an69 optional attribute needs the same default in the stub, the translation, and70 the creator.71- When a change reaches both frontends, update both. Model-specific code stays72 in that family's folder under `tensorrt_edgellm/models/` and73 `tensorrt_edgellm/builder/models/`.7475## 3. Implement the C++ plugin7677`cpp/CMakeLists.txt` globs `plugins/*.cpp` into `NvInfer_edgellm_plugin` and78recursively globs `kernels/*.cpp` plus `kernels/*.cu` into `edgellmKernels`79(`PLUGIN_JIT_*` sources go to `edgellmPluginJit` instead). A `.cu` file under80`cpp/plugins/` is silently not built: put CUDA code in `cpp/kernels/`. Kernel81artifacts and CuTe DSL groups belong to `edge-llm-cutedsl-kernel`.82831. Derive from `IPluginV3`, `IPluginV3OneCore`, `IPluginV3OneBuild`, and84 `IPluginV3OneRuntime`; return each from `getCapabilityInterface`. Register85 an `IPluginCreatorV3One` with `REGISTER_TENSORRT_PLUGIN` in namespace86 `trt_edgellm::plugins`, which is the C++ namespace only: the TensorRT plugin87 namespace stays empty (`""`). Keep name and version in named constants.882. Both builders create strongly typed networks (`cpp/builder/builderUtils.cpp`,89 `tensorrt_edgellm/builder/core/builder.py`). `getOutputDataTypes` must derive90 every output dtype from input dtypes and attributes, and91 `supportsFormatCombination` must accept exactly the contract.923. Build output shapes with `IExprBuilder`; require constant dims the kernel93 depends on. Validate min, opt, and max in `configurePlugin`, and revalidate94 in `onShapeChange` and `enqueue`. On violation, log which tensor, dtype, or95 shape is wrong and return `-1`; never throw across the library boundary.964. `getFieldsToSerialize` writes every attribute plus any build-derived state97 the runtime needs. The creator accepts that set in the runtime phase and98 returns `nullptr` for missing or unknown fields, or, with a log naming the99 missing group, when an optional kernel group is not compiled in.1005. Per-enqueue scratch goes through `getWorkspaceSize`. Persistent state is an101 engine binding or a runtime-owned pool. Weights are bindings prepared by the102 runtime. Do not allocate memory or repack weights in `enqueue`.1036. Launch only on the `stream` passed to `enqueue`; no default stream and no104 synchronous CUDA API. Check every CUDA return value.1057. Reuse `cpp/plugins/utils/pluginUtils.h` (`parsePluginScalarField`,106 `Serializer`, `kDEVICE_ALIGNMENT`) and `rt::Tensor` instead of adding helpers107 or raw owning pointers.1088. An output that aliases an input also needs builder gating, as109 `usesAliasedPluginIO` in `cpp/builder/llmBuilder.cpp` does for110 `DFlashTargetKVCacheUpdatePlugin` (plugin name `DFlashTargetKVCacheUpdate`).111112C++ builders and the example apps load the library through113`loadEdgellmPluginLib()` in `cpp/common/trtUtils.h`. Without `--plugin-path`, the direct builder uses the114installed package's plugin and only then `build/lib/libNvInfer_edgellm_plugin.so`,115so a stale installed library shadows a source build: pass `--plugin-path`.116117## 4. Tests118119Extend the nearest existing test; do not add a harness.120121| Contract | Extend |122|---|---|123| Stub semantics, emitted node, attributes, schema | the pattern in `tests/python-unittests/test_ple_pool_export.py` (not itself in the L0 list): eager output vs reference, `onnx.checker` with `check_custom_domain=True` |124| Creator, shape and type negotiation, execution, rejection | `PluginRunner` in `tests/python-unittests/test_plugin_base.py`, as `test_qkv_concat_plugin.py` uses it; `build` serializes and deserializes the engine, so execution also covers the runtime-phase creator; assert rejection with `expect_unsupported=True` |125| Direct lowering: name, input order, attributes | Fake-`Net` recording tests, patterned on `tests/python-unittests/test_experimental_attention_ragged_contract.py` (not in the L0 list) |126| Plugin or kernel internals in C++ | `unittests/cpp/plugins/<plugin>/` or `unittests/cpp/kernels/<dir>/`, registered in an `edgellm_add_unit_test` call in `unittests/CMakeLists.txt` (configure fails on unregistered sources); follow `edge-llm-cpp-test` |127128CI runs only tests named in `tests/test_lists/l0_python_ut.yml`, so list an added129Python test there; a GPU plugin test must also match `*plugin*.py` to run in the130device job. Cover every131dtype and rank the contract claims and at least one rejection path that132returns an actionable message. Mutation-test each added assertion as133`edge-llm-cpp-test` and `edge-llm-e2e-test` describe.134135## 5. End-to-end evidence136137For every claimed route, platform, and precision: export or direct-build a138representative model that enqueues the operation, build the engine, run139inference, and compare deterministically against the PyTorch reference.140Use `edge-llm-convert-model` to build and run it, `edge-llm-e2e-test` for the141model case, and `edge-llm-few-layer-test` when the change adds a layer, kernel,142or quantization recipe. Route accuracy failures to143`edge-llm-accuracy-debugging`.144145Not evidence: an ONNX file without an engine; a `PluginRunner` pass alone; a146collection-only run; a skipped platform; an engine built from a stale plugin147library.148149## Anti-patterns150151| Do not | Reason |152|---|---|153| Add, remove, or reorder plugin inputs when an attribute or existing input suffices | Every input change breaks both translations, the bindings, and all serialized engines |154| Reinterpret a serialized field or input without rebuilding engines | Old engines load and silently compute wrong values |155| Select output dtype from builder precision flags | Networks are strongly typed; the plugin owns its dtypes |156| Allocate, repack, or synchronize in `enqueue` | Hot-path rule in `AGENTS.md`; breaks CUDA graph capture and latency |157| Put creator names or `PluginField` construction in model files | Lowering is private to `tensorrt_edgellm/builder/ops/backend.py` and the translation table |158| Fall back silently when an optional kernel group is missing | The creator must fail with the missing group named; any fallback is explicit and logged |159160## Completion evidence161162Report the revision; the decision row from section 1 with its reason; files changed per layer; focused test commands with163pass, fail, and skip counts; and, for each route, platform, and precision, the164build and inference commands, GPU and SM, engine artifacts, reference165comparison, and verdict. State plainly what was not run and why.166167## Examples168169- "Add a fused operation used by the ONNX and checkpoint-direct builders": write the contract, wire both routes, implement the C++ plugin, then run unit and end-to-end tests.170- "Change an existing plugin's tensor layout": update the contract, every consumer, and the tests, then rebuild engines, because old engines load and silently compute wrong values.