1---
2name: triton-kernel-writing
3description: Write or review Triton kernels for vLLM, with practical guidance for generated-code inspection, launch grids, indexing, specialization, tuning, and representative performance validation.
4metadata:
5 source: https://skillmd.com/skills/vllm-project/triton-kernel-writing
6---
7
8# Triton Kernel Writing
9
10## Implementation
11
12- Follow the official
13 [Triton semantics](https://triton-lang.org/main/python-api/triton-semantics.html).
14 Check it when behavior may differ from Python or NumPy, especially type
15 promotion, integer division and modulo, casts, broadcasting, and variable
16 scoping.
17- Use the Triton kernel generated by `torch.compile` as a possible
18 implementation to inspect. Print Inductor's generated code with
19 `TORCH_LOGS="output_code" .venv/bin/python <script>` or enable
20 `torch._logging.set_logs(output_code=True)` before the compiled function
21 runs. Treat generated code as a reference, not as proof of correctness or
22 optimality.
23- Find reasonable defaults for compile-time knobs such as `BLOCK_SIZE`, or use
24 a small, legible heuristic when workloads need different choices. Use
25 `triton.autotune` only when tuning is critical to performance, such as for a
26 matrix multiplication. Otherwise prioritize simple code and fast startup.
27- Be careful to avoid unintended runtime JIT compilation. For example, put
28 unimportant runtime integer scalars in `do_not_specialize`, especially those
29 that may alternate between values such as 0 and 1, which can produce
30 different specialization keys.
31- The Triton compiler does not guarantee safe ordering when a kernel writes to
32 a pointer and subsequently reads from the same pointer. This pattern must
33 have a `tl.debug_barrier()` between the write and read. The barrier
34 synchronizes threads in the block; it does not synchronize separate program
35 instances.
36
37## Launch and Indexing
38
39- `grid[1]` and `grid[2]` must be at most 65,535. Choose or flatten the grid
40 order so those dimensions cannot exceed the limit for supported shapes.
41 For example, `num_tokens` is commonly 8K or 16K, but users may configure 32K
42 or more. If `num_tokens` is a grid dimension, it is safe to put it in
43 `grid[0]` (or tile it).
44- Use `int64` for offset arithmetic when an index can exceed 32-bit range,
45 especially for KV-cache addressing. Cast operands before multiplication or
46 addition so an intermediate does not overflow in 32-bit arithmetic.
47- A `[num_tokens, num_heads]` grid can be a good low-latency mapping for decode,
48 but it can be very slow for prefill. If the kernel serves prefill, consider
49 tiling tokens or otherwise increasing the work and locality per program.
50
51## Validation
52
53- Check correctness at boundary shapes and at sizes that exercise masks and
54 large offsets.
55- Choose accumulation and intermediate dtypes explicitly. Test numerically
56 difficult inputs, not only random, well-scaled tensors.
57- Use `$kernel-microbenchmark` for benchmark construction, measurement, and
58 interpretation.
59- Benchmark a sweep of `num_tokens` covering decode and representative prefill
60 workloads. Include relevant head counts and dimensions when they affect the
61 launch shape, and do not select an implementation or tuning heuristic from a
62 single setup.
63- Include compilation or autotuning overhead when evaluating startup behavior;
64 report steady-state kernel performance separately.