Writing TileLang Kernels
Pre-flight Checklist
Before writing any kernel:
- Verify the environment:
python -c "import tilelang; print(tilelang.__version__)"
- Identify the target operation: What are the input/output shapes, dtypes, and the mathematical operation?
- Write a PyTorch reference first: You will need this for correctness validation.
- Find the closest pattern — read
references/tilelang-examples.md for complete working examples:
| Pattern |
What to look for |
| GEMM |
T.gemm, shared memory, pipelining |
| GEMM + fusion |
ReLU/sigmoid epilogue after GEMM |
| Elementwise |
T.Parallel, shared memory staging |
| Reduction/Norm |
T.reduce_sum, two-pass pattern |
| Online Softmax |
Running max, exp, rescaling |
| Flash Attention |
Multi-fragment, softmax in loop |
| Linear Attention |
Chunked, running state |
For the complete TileLang language reference (primitives, control flow, instructions), read references/language-docs.md.
Kernel Anatomy
Every TileLang kernel has a two-level structure:
import tilelang
import tilelang.language as T
@tilelang.jit(out_idx=[-1]) # last arg is the output, returned by kernel call
def my_kernel(M, N, K, block_M, block_N, block_K, dtype=T.float16, accum_dtype=T.float32):
# Outer function: compile-time parameters (problem sizes, tile sizes, dtypes)
@T.prim_func
def kernel(
A: T.Tensor((M, K), dtype),
B: T.Tensor((K, N), dtype),
C: T.Tensor((M, N), dtype),
):
# Inner function: the actual GPU kernel
# T.Tensor declares buffer shapes and dtypes
with T.Kernel(T.ceildiv(N, block_N), T.ceildiv(M, block_M), threads=128) as (bx, by):
# Grid: (blocks_x, blocks_y), threads per block
# bx, by are block indices (like blockIdx.x, blockIdx.y)
# 1. Allocate on-chip memory
A_shared = T.alloc_shared((block_M, block_K), dtype) # shared memory (block-visible)
B_shared = T.alloc_shared((block_K, block_N), dtype) # shared memory
C_local = T.alloc_fragment((block_M, block_N), accum_dtype) # registers (per-thread)
# 2. Initialize
T.clear(C_local)
# 3. Compute (pipelined loop)
for ko in T.Pipelined(T.ceildiv(K, block_K), num_stages=3):
T.copy(A[by * block_M, ko * block_K], A_shared) # global -> shared
T.copy(B[ko * block_K, bx * block_N], B_shared) # global -> shared
T.gemm(A_shared, B_shared, C_local) # shared -> fragment
# 4. Write back
T.copy(C_local, C[by * block_M, bx * block_N]) # fragment -> global
return kernel # must return the inner function
Key Concepts
Memory scopes (3 tiers):
T.alloc_shared(shape, dtype) -- on-chip SRAM, visible to all threads in a block. Use for staging tiles from global memory.
T.alloc_fragment(shape, dtype) -- registers. Despite looking like full tile shapes, the compiler distributes across threads via layout inference. Use for accumulators and intermediate results.
T.alloc_local(shape, dtype) -- explicitly thread-private. Use for scalar accumulators.
Data movement: T.copy(src, dst) handles global<->shared<->fragment transfers. Extents are inferred from the destination buffer shape.
@tilelang.jit arguments:
out_idx=[-1] -- the last parameter is the output; kernel call returns it. out_idx=[2, 3] means params 2 and 3 are outputs.
- Without
out_idx -- caller must pre-allocate and pass all buffers; kernel returns None.
target="cuda" (optional) -- inferred automatically from input tensors if omitted.
T.Kernel grid:
T.Kernel(grid_x, grid_y, threads=N) -- 2D grid. T.Kernel(grid_x, grid_y, grid_z, threads=N) for 3D.
threads sets threads per block (typically 128 or 256).
- Use
T.ceildiv(dim, block_dim) for grid dimensions.
Step-by-Step Workflow
Write a PyTorch reference:
def ref_program(A, B):
return A @ B
Choose the kernel pattern: elementwise, reduction, GEMM, or fused.
Pick tile sizes (start with these defaults):
- GEMM:
block_M=128, block_N=128, block_K=32, num_stages=2, threads=128
- Elementwise:
block_M=64, block_N=64, threads=128
- Reduction:
block_M=64, block_N=128, threads=128 (reduce along N)
Write the kernel following the appropriate template (see below or read references/kernel-templates.md for full versions).
Compile and run:
kernel = my_kernel(M, N, K, block_M, block_N, block_K)
c = kernel(a, b) # with out_idx=[-1]
# OR
kernel(a, b, c) # without out_idx
Validate:
ref_c = ref_program(a, b)
torch.testing.assert_close(c, ref_c, rtol=1e-2, atol=1e-2)
Inspect generated CUDA (optional):
print(kernel.get_kernel_source())
Quick Templates
Elementwise (2D)
@tilelang.jit(out_idx=[-1])
def softplus(M, N, block_M, block_N, dtype=T.float16, accum_dtype=T.float32):
@T.prim_func
def kernel(X: T.Tensor((M, N), dtype), Y: T.Tensor((M, N), dtype)):
with T.Kernel(T.ceildiv(N, block_N), T.ceildiv(M, block_M), threads=128) as (bx, by):
X_shared = T.alloc_shared((block_M, block_N), dtype)
Y_local = T.alloc_fragment((block_M, block_N), accum_dtype)
T.copy(X[by * block_M, bx * block_N], X_shared)
for i, j in T.Parallel(block_M, block_N):
val = T.cast(X_shared[i, j], accum_dtype)
Y_local[i, j] = T.log(T.cast(1.0, accum_dtype) + T.exp(val))
T.copy(Y_local, Y[by * block_M, bx * block_N])
return kernel
GEMM (Minimal)
@tilelang.jit(out_idx=[-1])
def matmul(M, N, K, block_M, block_N, block_K, dtype=T.float16, accum_dtype=T.float32):
@T.prim_func
def kernel(A: T.Tensor((M, K), dtype), B: T.Tensor((K, N), dtype), C: T.Tensor((M, N), dtype)):
with T.Kernel(T.ceildiv(N, block_N), T.ceildiv(M, block_M), threads=128) as (bx, by):
A_shared = T.alloc_shared((block_M, block_K), dtype)
B_shared = T.alloc_shared((block_K, block_N), dtype)
C_local = T.alloc_fragment((block_M, block_N), accum_dtype)
T.clear(C_local)
for ko in T.Pipelined(T.ceildiv(K, block_K), num_stages=3):
T.copy(A[by * block_M, ko * block_K], A_shared)
T.copy(B[ko * block_K, bx * block_N], B_shared)
T.gemm(A_shared, B_shared, C_local)
T.copy(C_local, C[by * block_M, bx * block_N])
return kernel
For more templates (1D elementwise, row reduction, dynamic shape GEMM) with complete imports, host code, and validation, read references/kernel-templates.md. For full working examples, read references/tilelang-examples.md.
Key API Patterns
T.ceildiv(a, b) -- ceiling division for grid dimensions
T.clear(buf) / T.fill(buf, value) -- initialize buffers
T.reduce_sum(src, dst, dim=N), T.reduce_max(...), T.reduce_min(...) -- tile-level reductions
T.exp, T.log, T.exp2, T.log2, T.rsqrt, T.sigmoid, T.max, T.min -- elementwise math
T.cast(value, dtype) -- explicit type casting
T.if_then_else(cond, true_val, false_val) -- conditional expression (for masking)
T.atomic_add(dst, value) -- atomic addition to global memory
T.infinity(dtype) -- typed infinity constant (use -T.infinity(dtype) for init)
T.gemm(A_shared, B_shared, C_fragment, transpose_A=False, transpose_B=False) -- tensor core GEMM
T.Pipelined(iters, num_stages=N) -- software pipelining (overlaps copy + compute)
T.Parallel(dim0, dim1) -- parallel loops mapped to threads
T.serial(start, stop) -- sequential loop
T.use_swizzle(panel_size=10, enable=True) -- L2 cache locality optimization
For the complete instruction reference, read references/api-quick-reference.md.
Common Pitfalls
| Symptom |
Cause |
Fix |
| Garbage/NaN in output |
Missing T.clear(C_local) before accumulation |
Add T.clear(C_local) before the pipelined loop |
T.gemm K shape check failed: K_A = X, K_B = Y |
Shape mismatch in T.gemm args |
A must be (BM,BK), B must be (BK,BN), C must be (BM,BN) |
| Output has random-looking values (not zeros) |
Missing T.copy(result, Output[...]) writeback |
Add the final T.copy from fragment to global memory |
Kernel expected N inputs, but M are provided |
Wrong arg count due to out_idx |
With out_idx=[-1]: pass N-1 args, kernel returns output. Without: pass all N args. |
Invalid TMA descriptor: globalStrides must be multiple of 16 bytes |
Non-aligned inner dimension on Blackwell/Hopper |
For fp16: inner dims must be multiples of 8. For fp32: multiples of 4. |
| Kernel compiles but hangs |
Infinite loop or deadlock |
Check T.Pipelined iteration count is correct: T.ceildiv(K, block_K) not K |
Escalation
- Kernel compiles but produces wrong results → use the debugging-tilelang-programs skill
- Kernel is correct but slow → use the profiling-tilelang-programs skill, then optimizing-tilelang-programs
- Need forward + backward passes → use the testing-fwd-bwd-kernels skill
1---2name: writing-tilelang-kernels3description: >4 How to write TileLang GPU kernels from scratch or by adapting existing patterns.5 Use this skill whenever the user wants to create a new TileLang kernel, implement6 a GPU operator in TileLang, write a prim_func, set up tiling/blocking, define7 shared memory and fragment allocations, use T.gemm or T.copy, create a JIT-compiled8 kernel, or port an algorithm (GEMM, elementwise, reduction, attention, normalization,9 softmax, convolution) to TileLang. Also use when the user asks about tilelang.jit,10 T.prim_func, T.Kernel, T.Tensor, T.alloc_shared, T.alloc_fragment, T.Pipelined,11 T.Parallel, or any TileLang DSL construct for kernel authoring. Even for questions12 like "how do I start with TileLang" or "show me a basic TileLang example", this13 skill provides the right templates and workflow.14metadata:15 source: https://skillmd.com/skills/wenyi-li/writing-tilelang-kernels16---1718# Writing TileLang Kernels1920## Pre-flight Checklist2122Before writing any kernel:23241. **Verify the environment**: `python -c "import tilelang; print(tilelang.__version__)"`252. **Identify the target operation**: What are the input/output shapes, dtypes, and the mathematical operation?263. **Write a PyTorch reference first**: You will need this for correctness validation.274. **Find the closest pattern** — read `references/tilelang-examples.md` for complete working examples:2829| Pattern | What to look for |30|---------|-----------------|31| GEMM | T.gemm, shared memory, pipelining |32| GEMM + fusion | ReLU/sigmoid epilogue after GEMM |33| Elementwise | T.Parallel, shared memory staging |34| Reduction/Norm | T.reduce_sum, two-pass pattern |35| Online Softmax | Running max, exp, rescaling |36| Flash Attention | Multi-fragment, softmax in loop |37| Linear Attention | Chunked, running state |3839For the complete TileLang language reference (primitives, control flow, instructions), read `references/language-docs.md`.4041## Kernel Anatomy4243Every TileLang kernel has a two-level structure:4445```python46import tilelang47import tilelang.language as T4849@tilelang.jit(out_idx=[-1]) # last arg is the output, returned by kernel call50def my_kernel(M, N, K, block_M, block_N, block_K, dtype=T.float16, accum_dtype=T.float32):51 # Outer function: compile-time parameters (problem sizes, tile sizes, dtypes)5253 @T.prim_func54 def kernel(55 A: T.Tensor((M, K), dtype),56 B: T.Tensor((K, N), dtype),57 C: T.Tensor((M, N), dtype),58 ):59 # Inner function: the actual GPU kernel60 # T.Tensor declares buffer shapes and dtypes6162 with T.Kernel(T.ceildiv(N, block_N), T.ceildiv(M, block_M), threads=128) as (bx, by):63 # Grid: (blocks_x, blocks_y), threads per block64 # bx, by are block indices (like blockIdx.x, blockIdx.y)6566 # 1. Allocate on-chip memory67 A_shared = T.alloc_shared((block_M, block_K), dtype) # shared memory (block-visible)68 B_shared = T.alloc_shared((block_K, block_N), dtype) # shared memory69 C_local = T.alloc_fragment((block_M, block_N), accum_dtype) # registers (per-thread)7071 # 2. Initialize72 T.clear(C_local)7374 # 3. Compute (pipelined loop)75 for ko in T.Pipelined(T.ceildiv(K, block_K), num_stages=3):76 T.copy(A[by * block_M, ko * block_K], A_shared) # global -> shared77 T.copy(B[ko * block_K, bx * block_N], B_shared) # global -> shared78 T.gemm(A_shared, B_shared, C_local) # shared -> fragment7980 # 4. Write back81 T.copy(C_local, C[by * block_M, bx * block_N]) # fragment -> global8283 return kernel # must return the inner function84```8586### Key Concepts8788**Memory scopes** (3 tiers):89- `T.alloc_shared(shape, dtype)` -- on-chip SRAM, visible to all threads in a block. Use for staging tiles from global memory.90- `T.alloc_fragment(shape, dtype)` -- registers. Despite looking like full tile shapes, the compiler distributes across threads via layout inference. Use for accumulators and intermediate results.91- `T.alloc_local(shape, dtype)` -- explicitly thread-private. Use for scalar accumulators.9293**Data movement**: `T.copy(src, dst)` handles global<->shared<->fragment transfers. Extents are inferred from the destination buffer shape.9495**`@tilelang.jit` arguments**:96- `out_idx=[-1]` -- the last parameter is the output; kernel call returns it. `out_idx=[2, 3]` means params 2 and 3 are outputs.97- Without `out_idx` -- caller must pre-allocate and pass all buffers; kernel returns None.98- `target="cuda"` (optional) -- inferred automatically from input tensors if omitted.99100**`T.Kernel` grid**:101- `T.Kernel(grid_x, grid_y, threads=N)` -- 2D grid. `T.Kernel(grid_x, grid_y, grid_z, threads=N)` for 3D.102- `threads` sets threads per block (typically 128 or 256).103- Use `T.ceildiv(dim, block_dim)` for grid dimensions.104105## Step-by-Step Workflow1061071. **Write a PyTorch reference**:108 ```python109 def ref_program(A, B):110 return A @ B111 ```1121132. **Choose the kernel pattern**: elementwise, reduction, GEMM, or fused.1141153. **Pick tile sizes** (start with these defaults):116 - GEMM: `block_M=128, block_N=128, block_K=32, num_stages=2, threads=128`117 - Elementwise: `block_M=64, block_N=64, threads=128`118 - Reduction: `block_M=64, block_N=128, threads=128` (reduce along N)1191204. **Write the kernel** following the appropriate template (see below or read `references/kernel-templates.md` for full versions).1211225. **Compile and run**:123 ```python124 kernel = my_kernel(M, N, K, block_M, block_N, block_K)125 c = kernel(a, b) # with out_idx=[-1]126 # OR127 kernel(a, b, c) # without out_idx128 ```1291306. **Validate**:131 ```python132 ref_c = ref_program(a, b)133 torch.testing.assert_close(c, ref_c, rtol=1e-2, atol=1e-2)134 ```1351367. **Inspect generated CUDA** (optional):137 ```python138 print(kernel.get_kernel_source())139 ```140141## Quick Templates142143### Elementwise (2D)144145```python146@tilelang.jit(out_idx=[-1])147def softplus(M, N, block_M, block_N, dtype=T.float16, accum_dtype=T.float32):148 @T.prim_func149 def kernel(X: T.Tensor((M, N), dtype), Y: T.Tensor((M, N), dtype)):150 with T.Kernel(T.ceildiv(N, block_N), T.ceildiv(M, block_M), threads=128) as (bx, by):151 X_shared = T.alloc_shared((block_M, block_N), dtype)152 Y_local = T.alloc_fragment((block_M, block_N), accum_dtype)153 T.copy(X[by * block_M, bx * block_N], X_shared)154 for i, j in T.Parallel(block_M, block_N):155 val = T.cast(X_shared[i, j], accum_dtype)156 Y_local[i, j] = T.log(T.cast(1.0, accum_dtype) + T.exp(val))157 T.copy(Y_local, Y[by * block_M, bx * block_N])158 return kernel159```160161### GEMM (Minimal)162163```python164@tilelang.jit(out_idx=[-1])165def matmul(M, N, K, block_M, block_N, block_K, dtype=T.float16, accum_dtype=T.float32):166 @T.prim_func167 def kernel(A: T.Tensor((M, K), dtype), B: T.Tensor((K, N), dtype), C: T.Tensor((M, N), dtype)):168 with T.Kernel(T.ceildiv(N, block_N), T.ceildiv(M, block_M), threads=128) as (bx, by):169 A_shared = T.alloc_shared((block_M, block_K), dtype)170 B_shared = T.alloc_shared((block_K, block_N), dtype)171 C_local = T.alloc_fragment((block_M, block_N), accum_dtype)172 T.clear(C_local)173 for ko in T.Pipelined(T.ceildiv(K, block_K), num_stages=3):174 T.copy(A[by * block_M, ko * block_K], A_shared)175 T.copy(B[ko * block_K, bx * block_N], B_shared)176 T.gemm(A_shared, B_shared, C_local)177 T.copy(C_local, C[by * block_M, bx * block_N])178 return kernel179```180181For more templates (1D elementwise, row reduction, dynamic shape GEMM) with complete imports, host code, and validation, read `references/kernel-templates.md`. For full working examples, read `references/tilelang-examples.md`.182183## Key API Patterns184185- `T.ceildiv(a, b)` -- ceiling division for grid dimensions186- `T.clear(buf)` / `T.fill(buf, value)` -- initialize buffers187- `T.reduce_sum(src, dst, dim=N)`, `T.reduce_max(...)`, `T.reduce_min(...)` -- tile-level reductions188- `T.exp`, `T.log`, `T.exp2`, `T.log2`, `T.rsqrt`, `T.sigmoid`, `T.max`, `T.min` -- elementwise math189- `T.cast(value, dtype)` -- explicit type casting190- `T.if_then_else(cond, true_val, false_val)` -- conditional expression (for masking)191- `T.atomic_add(dst, value)` -- atomic addition to global memory192- `T.infinity(dtype)` -- typed infinity constant (use `-T.infinity(dtype)` for init)193- `T.gemm(A_shared, B_shared, C_fragment, transpose_A=False, transpose_B=False)` -- tensor core GEMM194- `T.Pipelined(iters, num_stages=N)` -- software pipelining (overlaps copy + compute)195- `T.Parallel(dim0, dim1)` -- parallel loops mapped to threads196- `T.serial(start, stop)` -- sequential loop197- `T.use_swizzle(panel_size=10, enable=True)` -- L2 cache locality optimization198199For the complete instruction reference, read `references/api-quick-reference.md`.200201## Common Pitfalls202203| Symptom | Cause | Fix |204|---------|-------|-----|205| Garbage/NaN in output | Missing `T.clear(C_local)` before accumulation | Add `T.clear(C_local)` before the pipelined loop |206| `T.gemm K shape check failed: K_A = X, K_B = Y` | Shape mismatch in T.gemm args | A must be (BM,BK), B must be (BK,BN), C must be (BM,BN) |207| Output has random-looking values (not zeros) | Missing `T.copy(result, Output[...])` writeback | Add the final T.copy from fragment to global memory |208| `Kernel expected N inputs, but M are provided` | Wrong arg count due to `out_idx` | With `out_idx=[-1]`: pass N-1 args, kernel returns output. Without: pass all N args. |209| `Invalid TMA descriptor: globalStrides must be multiple of 16 bytes` | Non-aligned inner dimension on Blackwell/Hopper | For fp16: inner dims must be multiples of 8. For fp32: multiples of 4. |210| Kernel compiles but hangs | Infinite loop or deadlock | Check `T.Pipelined` iteration count is correct: `T.ceildiv(K, block_K)` not `K` |211212## Escalation213214- Kernel compiles but produces wrong results → use the **debugging-tilelang-programs** skill215- Kernel is correct but slow → use the **profiling-tilelang-programs** skill, then **optimizing-tilelang-programs**216- Need forward + backward passes → use the **testing-fwd-bwd-kernels** skill