Edge-LLM CuTe DSL kernels
A CuTe DSL kernel reaches the runtime through a chain of files that must agree
on variant names, supported SMs, and the exported ABI. CMake never runs Python.
It links whatever is in cpp/kernels/cuteDSLArtifact/<arch>/<tag>/, and that
can be an extracted prebuilt that does not contain your change.
Read these before proposing an edit. They are canonical, so do not restate them:
kernelSrcs/README.md: dependencies, Docker and venv generation, script
flags, CMake integration, cross-compilation, and the tarball matrix.
kernelSrcs/<group>/README.md, when present: variants, ABI, and standalone tests.
kernelSrcs/build_cutedsl.py: KernelVariant, KERNEL_VARIANTS, _compile_command.
cmake/CuteDsl.cmake: cute_dsl_setup and every group or variant define.
CODING_GUIDELINES.md and the CUDA rules in AGENTS.md, for runner code.
The cross-file wiring map, with the edit order for an added variant or group, is in
references/wiring-map.md. Read it before your first
edit.
Before designing a kernel, read one recent merged kernel or variant addition
(git log --oneline --no-merges -n 10 origin/main -- kernelSrcs cmake/CuteDsl.cmake)
instead of copying the oldest similar file. Repository history is authoritative
for the current registration and test shape; this skill is the invariant checklist.
Instructions
Follow steps 1 to 5 in order: pin the target and baseline, develop the kernel, regenerate artifacts and reconfigure, test against the regenerated artifact, and package only after validation. Report with Completion evidence.
1. Pin the target
Record these values first. Each one selects a different artifact.
| Input |
Where it is consumed |
| Target SM or SM set |
--gpu_arch sm_NN[,sm_MM] sets the artifact tag sm_NN or sm_NN_sm_MM |
| CUDA major and compiler flavor |
--cuda-version; one venv per nvidia-cutlass-dsl[cu12] or [cu13]; tarball suffix _cudaNN |
| Target CPU arch |
--arch x86_64 or aarch64; an arch that differs from the host is a cross build |
| Runtime platform |
CMake EMBEDDED_TARGET or CUTE_DSL_ARTIFACT_TAG; see docs/source/user_guide/getting_started/support-matrix.md |
| Validation device |
A GPU of every SM you claim. Without one, report the run as blocked. |
Claim only the SMs you can validate. supported_sms is the build whitelist,
and the group README's variant table is the user-facing claim.
2. Develop the kernel
| Loop |
What it proves |
What it does not prove |
| JIT standalone script on a GPU |
Kernel math on that GPU's SM |
The AOT trace, the exported ABI, or other SMs |
AOT export on a CPU host, through build_cutedsl.py |
The variant traces and exports for --gpu_arch with storage-free placeholders |
Any numerical result |
| C++ test linked against the fresh artifact |
ABI, runner dispatch, and numerics on the device's SM |
Model integration |
- Keep JIT and AOT on the same kernel class; see the script patterns in the wiring map.
- AOT traces use
kernelSrcs/cutedsl_utils/aot_placeholders.py. Each trace
dimension is either baked into the cubin or marked dynamic. Name each baked
value as a constant and enforce the same limit in the runner's admission check.
- Never query the device in a kernel script; the build host's GPU is not the
target. A script that needs the SM reads
--target_sm; its registry entry
sets wants_target_sm=True.
- Selection is per group, not per variant. When the consumer is the attention
stack, follow the group README and include its required FMHA group in a fresh
artifact. Do not add
fmha automatically for an unrelated standalone group.
3. Generate fresh artifacts and configure
- Generate with the Docker or venv workflow in
kernelSrcs/README.md. Use
--clean for a first build, after you add, rename, or remove a variant, after
you change supported_sms, and before packaging. A single-SM build without
--clean merges into metadata.json (union of variant lists), so stale
members survive. Multi-SM and Windows artifacts refuse to rebuild into an existing artifact without --clean.
- Cross build:
--arch aarch64 on an x86_64 host emits target host objects and
embeds the target-arch runtime shim. This proves the build only; run the
tests on the target device. A native build on the device skips the cross step.
- Reconfigure CMake. In the configure log, confirm that it did not print
CuTe DSL: extracting prebuilt and that it printed the CuTe DSL: ... <DEFINE> set
status line for each variant-specific define you added.
- Inspect
metadata.json: variants lists your object stems (equal to the variant names unless the script suffixes the export), gpu_arch or
gpu_archs matches the target, and build_date is later than your last source edit.
A prebuilt tarball under kernelSrcs/cuteDSLPrebuilt/, or an artifact
directory extracted from one, is never evidence for modified kernel source.
4. Test
Extend the existing tests next to the group. Do not add a test framework.
| Layer |
Existing home |
Requirement |
| Registry and command contract (CPU) |
tests/python-unittests/test_build_cutedsl_helpers.py (in the L0 list; test_cutedsl_aot.py is not, and needs cutlass) |
Variant names, the SM whitelist, and rejected SMs, following the per-group tests already there |
| Kernel numerics (GPU) |
The group's GTest under unittests/, for example unittests/cpp/kernels/moe/f16MoeCuteDslTests.cu |
A human-readable reference, every exported variant, each supported dtype, boundary and residual shapes, and empty or zero-length inputs where the ABI allows them |
| Plugin or runtime integration |
The consumer's test, for example tests/python-unittests/test_fp16_moe_plugin.py |
Required whenever the kernel is not standalone; ownership moves to edge-llm-plugin-extension |
| Model path |
edge-llm-e2e-test, plus edge-llm-few-layer-test for an added kernel |
Only when the change affects model output |
- Follow
edge-llm-cpp-test for GTest structure and mutation checks.
- Run the numeric test on each claimed SM. Name failures by variant
(
<variant> at sm_NN, with the dtype and shape) rather than by group.
- Record the workload (shapes, dtype, batch), the GPU and SM, and the CUDA
flavor with every performance claim. Do not write numbers into this skill or
into a README.
5. Package after validation
Package only after sections 3 and 4 pass on every claimed SM.
- Update the group README variant table and its verified SMs. For an added group,
also update the group lists in
kernelSrcs/README.md.
- Add or change a matrix target in
kernelSrcs/Dockerfile.cutedsl
(CUTE_DSL_MATRIX) and in CUTE_DSL_TARGETS in kernelSrcs/build_cutedsl_tarballs.sh (its per-CUDA-major default, used when CUTE_DSL_MATRIX is unset).
Mirror the target in the tarball list in kernelSrcs/README.md and in the
matrix test in tests/python-unittests/test_build_cutedsl_helpers.py.
- Produce tarballs and
.sha256 files with the Docker builder, which validates
metadata, ELF arch, layout, and checksum. Tarballs are git-ignored; never
commit them.
- Borrowed kernel code keeps its upstream license header and adds an
Adapted from <project> <version> <path> line, as
kernelSrcs/f16_moe_cutedsl/grouped_gemm_blackwell.py does. The group README
states the origin. CONTRIBUTING.md requires DCO sign-off that certifies the
right to submit, and .pre-commit-config.yaml excludes kernelSrcs/, so
no hook adds or checks a header there.
Anti-patterns
| Do not |
Reason |
Gate a variant with a family-wide define such as CUTE_DSL_FMHA_BLACKWELL_ENABLED |
That define keys on fmha_d64, which only SM100/101/103/110 build. Code under it silently disappears on SM120/121. Key a define on your own variant name. |
| Dispatch on SM family alone |
A single-SM artifact holds one SM's cubins. Also check the artifact SM, as isArtifactSm does in cuteDslF16MoeRunner.cpp. |
Rename the export in the script or ignore --file_name/--function_prefix |
CMake detection, the metadata variants, and multi-SM __arch<NN> symbol dispatch all key on the emitted object stem. Suffix it only deliberately, as gdn_decode_mtp does, and use the stem in CMake and C++. |
Widen supported_sms without a run on that SM |
The README and plugin admission advertise a kernel that nobody executed. |
| Validate with a JIT run or a prebuilt only |
Neither exercises the AOT artifact your source produces. |
Allocate, use the default stream, or skip error checks in run() |
Follow CODING_GUIDELINES.md and the CUDA rules in AGENTS.md: workspace from the caller, an explicit cudaStream_t, and checked CUDA calls. |
| Add an environment variable to pick a variant |
Selection belongs in the registry, the defines, and runner arguments. |
Completion evidence
Report:
- The revision and the exact
build_cutedsl.py and CMake commands.
- The
metadata.json fields gpu_arch/gpu_archs, variants,
cutlass_dsl_version, cuda_version, and build_date.
- The configure lines that show each define.
- Test commands with pass, fail, and skip counts per SM, dtype, and variant.
- The GPU used for each SM, and whether the artifact was a native or cross build.
Name each claimed SM that was not executed and the reason. Prebuilt artifacts,
JIT-only runs, export-only runs, and collection-only runs are not evidence for
changed kernel source.
Examples
- "Change a kernel tile size and verify it": develop the change, regenerate the artifact for the target SM, rebuild, and run the kernel tests plus an end-to-end check.
- "Add a new variant for another SM": register the variant and every dependent define, regenerate, and show the runtime links the new artifact.
1---2name: edge-llm-cutedsl-kernel3license: Apache-2.04metadata:5 source: https://skillmd.com/skills/nvidia/edge-llm-cutedsl-kernel6 author: TensorRT Edge-LLM Team <noreply@nvidia.com>7 tags: [edge-llm, cutedsl, kernel, artifacts]8description: Add or change a TensorRT Edge-LLM CuTe DSL kernel end to end, from the Python kernel source through the KERNEL_VARIANTS registry in kernelSrcs/build_cutedsl.py, GPU-free AOT artifact generation (native or AArch64 cross build), the metadata-driven CUTE_DSL_*_ENABLED defines in cmake/CuteDsl.cmake, and the C++ runner and SM dispatch, to numerical tests on every claimed SM and prebuilt tarball packaging. Use when adding a CuTe DSL kernel group or variant, changing supported_sms, regenerating cpp/kernels/cuteDSLArtifact, debugging a missing CuTe DSL define, symbol, or module-load failure, or preparing cutedsl tarballs. Do not use it to create or register a TensorRT plugin (use edge-llm-plugin-extension), for GTest authoring rules (use edge-llm-cpp-test), or for model deployment tests (use edge-llm-e2e-test).9---1011<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->12<!-- SPDX-License-Identifier: Apache-2.0 -->1314# Edge-LLM CuTe DSL kernels1516A CuTe DSL kernel reaches the runtime through a chain of files that must agree17on variant names, supported SMs, and the exported ABI. CMake never runs Python.18It links whatever is in `cpp/kernels/cuteDSLArtifact/<arch>/<tag>/`, and that19can be an extracted prebuilt that does not contain your change.2021Read these before proposing an edit. They are canonical, so do not restate them:2223- `kernelSrcs/README.md`: dependencies, Docker and venv generation, script24 flags, CMake integration, cross-compilation, and the tarball matrix.25- `kernelSrcs/<group>/README.md`, when present: variants, ABI, and standalone tests.26- `kernelSrcs/build_cutedsl.py`: `KernelVariant`, `KERNEL_VARIANTS`, `_compile_command`.27- `cmake/CuteDsl.cmake`: `cute_dsl_setup` and every group or variant define.28- `CODING_GUIDELINES.md` and the CUDA rules in `AGENTS.md`, for runner code.2930The cross-file wiring map, with the edit order for an added variant or group, is in31[references/wiring-map.md](references/wiring-map.md). Read it before your first32edit.3334Before designing a kernel, read one recent merged kernel or variant addition35(`git log --oneline --no-merges -n 10 origin/main -- kernelSrcs cmake/CuteDsl.cmake`)36instead of copying the oldest similar file. Repository history is authoritative37for the current registration and test shape; this skill is the invariant checklist.3839## Instructions4041Follow steps 1 to 5 in order: pin the target and baseline, develop the kernel, regenerate artifacts and reconfigure, test against the regenerated artifact, and package only after validation. Report with Completion evidence.4243## 1. Pin the target4445Record these values first. Each one selects a different artifact.4647| Input | Where it is consumed |48|---|---|49| Target SM or SM set | `--gpu_arch sm_NN[,sm_MM]` sets the artifact tag `sm_NN` or `sm_NN_sm_MM` |50| CUDA major and compiler flavor | `--cuda-version`; one venv per `nvidia-cutlass-dsl[cu12]` or `[cu13]`; tarball suffix `_cudaNN` |51| Target CPU arch | `--arch x86_64` or `aarch64`; an arch that differs from the host is a cross build |52| Runtime platform | CMake `EMBEDDED_TARGET` or `CUTE_DSL_ARTIFACT_TAG`; see `docs/source/user_guide/getting_started/support-matrix.md` |53| Validation device | A GPU of every SM you claim. Without one, report the run as blocked. |5455Claim only the SMs you can validate. `supported_sms` is the build whitelist,56and the group README's variant table is the user-facing claim.5758## 2. Develop the kernel5960| Loop | What it proves | What it does not prove |61|---|---|---|62| JIT standalone script on a GPU | Kernel math on that GPU's SM | The AOT trace, the exported ABI, or other SMs |63| AOT export on a CPU host, through `build_cutedsl.py` | The variant traces and exports for `--gpu_arch` with storage-free placeholders | Any numerical result |64| C++ test linked against the fresh artifact | ABI, runner dispatch, and numerics on the device's SM | Model integration |6566- Keep JIT and AOT on the same kernel class; see the script patterns in the wiring map.67- AOT traces use `kernelSrcs/cutedsl_utils/aot_placeholders.py`. Each trace68 dimension is either baked into the cubin or marked dynamic. Name each baked69 value as a constant and enforce the same limit in the runner's admission check.70- Never query the device in a kernel script; the build host's GPU is not the71 target. A script that needs the SM reads `--target_sm`; its registry entry72 sets `wants_target_sm=True`.73- Selection is per group, not per variant. When the consumer is the attention74 stack, follow the group README and include its required FMHA group in a fresh75 artifact. Do not add `fmha` automatically for an unrelated standalone group.7677## 3. Generate fresh artifacts and configure78791. Generate with the Docker or venv workflow in `kernelSrcs/README.md`. Use80 `--clean` for a first build, after you add, rename, or remove a variant, after81 you change `supported_sms`, and before packaging. A single-SM build without82 `--clean` merges into `metadata.json` (union of variant lists), so stale83 members survive. Multi-SM and Windows artifacts refuse to rebuild into an existing artifact without `--clean`.842. Cross build: `--arch aarch64` on an x86_64 host emits target host objects and85 embeds the target-arch runtime shim. This proves the build only; run the86 tests on the target device. A native build on the device skips the cross step.873. Reconfigure CMake. In the configure log, confirm that it did not print88 `CuTe DSL: extracting prebuilt` and that it printed the `CuTe DSL: ... <DEFINE> set`89 status line for each variant-specific define you added.904. Inspect `metadata.json`: `variants` lists your object stems (equal to the variant names unless the script suffixes the export), `gpu_arch` or91 `gpu_archs` matches the target, and `build_date` is later than your last source edit.9293A prebuilt tarball under `kernelSrcs/cuteDSLPrebuilt/`, or an artifact94directory extracted from one, is never evidence for modified kernel source.9596## 4. Test9798Extend the existing tests next to the group. Do not add a test framework.99100| Layer | Existing home | Requirement |101|---|---|---|102| Registry and command contract (CPU) | `tests/python-unittests/test_build_cutedsl_helpers.py` (in the L0 list; `test_cutedsl_aot.py` is not, and needs `cutlass`) | Variant names, the SM whitelist, and rejected SMs, following the per-group tests already there |103| Kernel numerics (GPU) | The group's GTest under `unittests/`, for example `unittests/cpp/kernels/moe/f16MoeCuteDslTests.cu` | A human-readable reference, every exported variant, each supported dtype, boundary and residual shapes, and empty or zero-length inputs where the ABI allows them |104| Plugin or runtime integration | The consumer's test, for example `tests/python-unittests/test_fp16_moe_plugin.py` | Required whenever the kernel is not standalone; ownership moves to `edge-llm-plugin-extension` |105| Model path | `edge-llm-e2e-test`, plus `edge-llm-few-layer-test` for an added kernel | Only when the change affects model output |106107- Follow `edge-llm-cpp-test` for GTest structure and mutation checks.108- Run the numeric test on each claimed SM. Name failures by variant109 (`<variant>` at `sm_NN`, with the dtype and shape) rather than by group.110- Record the workload (shapes, dtype, batch), the GPU and SM, and the CUDA111 flavor with every performance claim. Do not write numbers into this skill or112 into a README.113114## 5. Package after validation115116Package only after sections 3 and 4 pass on every claimed SM.1171181. Update the group README variant table and its verified SMs. For an added group,119 also update the group lists in `kernelSrcs/README.md`.1202. Add or change a matrix target in `kernelSrcs/Dockerfile.cutedsl`121 (`CUTE_DSL_MATRIX`) and in `CUTE_DSL_TARGETS` in `kernelSrcs/build_cutedsl_tarballs.sh` (its per-CUDA-major default, used when `CUTE_DSL_MATRIX` is unset).122 Mirror the target in the tarball list in `kernelSrcs/README.md` and in the123 matrix test in `tests/python-unittests/test_build_cutedsl_helpers.py`.1243. Produce tarballs and `.sha256` files with the Docker builder, which validates125 metadata, ELF arch, layout, and checksum. Tarballs are git-ignored; never126 commit them.1274. Borrowed kernel code keeps its upstream license header and adds an128 `Adapted from <project> <version> <path>` line, as129 `kernelSrcs/f16_moe_cutedsl/grouped_gemm_blackwell.py` does. The group README130 states the origin. `CONTRIBUTING.md` requires DCO sign-off that certifies the131 right to submit, and `.pre-commit-config.yaml` excludes `kernelSrcs/`, so132 no hook adds or checks a header there.133134## Anti-patterns135136| Do not | Reason |137|---|---|138| Gate a variant with a family-wide define such as `CUTE_DSL_FMHA_BLACKWELL_ENABLED` | That define keys on `fmha_d64`, which only SM100/101/103/110 build. Code under it silently disappears on SM120/121. Key a define on your own variant name. |139| Dispatch on SM family alone | A single-SM artifact holds one SM's cubins. Also check the artifact SM, as `isArtifactSm` does in `cuteDslF16MoeRunner.cpp`. |140| Rename the export in the script or ignore `--file_name`/`--function_prefix` | CMake detection, the metadata `variants`, and multi-SM `__arch<NN>` symbol dispatch all key on the emitted object stem. Suffix it only deliberately, as `gdn_decode_mtp` does, and use the stem in CMake and C++. |141| Widen `supported_sms` without a run on that SM | The README and plugin admission advertise a kernel that nobody executed. |142| Validate with a JIT run or a prebuilt only | Neither exercises the AOT artifact your source produces. |143| Allocate, use the default stream, or skip error checks in `run()` | Follow `CODING_GUIDELINES.md` and the CUDA rules in `AGENTS.md`: workspace from the caller, an explicit `cudaStream_t`, and checked CUDA calls. |144| Add an environment variable to pick a variant | Selection belongs in the registry, the defines, and runner arguments. |145146## Completion evidence147148Report:149150- The revision and the exact `build_cutedsl.py` and CMake commands.151- The `metadata.json` fields `gpu_arch`/`gpu_archs`, `variants`,152 `cutlass_dsl_version`, `cuda_version`, and `build_date`.153- The configure lines that show each define.154- Test commands with pass, fail, and skip counts per SM, dtype, and variant.155- The GPU used for each SM, and whether the artifact was a native or cross build.156157Name each claimed SM that was not executed and the reason. Prebuilt artifacts,158JIT-only runs, export-only runs, and collection-only runs are not evidence for159changed kernel source.160161## Examples162163- "Change a kernel tile size and verify it": develop the change, regenerate the artifact for the target SM, rebuild, and run the kernel tests plus an end-to-end check.164- "Add a new variant for another SM": register the variant and every dependent define, regenerate, and show the runtime links the new artifact.