TileLang Layout System
Core types (src/layout/layout.h, Python tilelang/layout/)
tl::Layout maps logical coordinates to physical coordinates:
input_size_ (logical shape) + forward_index_ (one PrimExpr per output
axis). Expressions are written in process-wide placeholder Vars
InputPlaceholder(i) (printed _i, _j, ...). Key APIs:
InputShape/OutputShape/GetForwardIndex/Forward(vars)/Inverse/ InverseWithLevel/Reshape/DetectInjective/DebugOutput. OutputShape() is
DERIVED from the expressions' ranges by the Analyzer, not stored.
tl::Fragment : Layout adds a thread dimension for register buffers:
forward_thread_ (PrimExpr, may reference ReplicationPlaceholder(),
printed _rep) and replicate_size_. ThreadExtent() =
max(forward_thread)+1 under bound domains — also derived, not stored.
thread_range_ (via BindThreadRange) offsets the thread space for warp
specialization; the forward map itself stays in normalized [0, T).
Replication semantics: each logical point is held by replicate_size
threads; Replicate/DeReplicate/CondenseReplicateVar manipulate the rep
axis.
Gotchas (verified, easy to trip on):
FragmentNode::GetForwardVars() PREPENDS ReplicationPlaceholder()
when replicate_size > 1 (src/layout/layout.cc:1103). The input
placeholders are always the TRAILING InputDim() entries. Python code
that zips get_forward_vars() against the shape positionally is wrong
for replicated fragments.
- The canonical (thread, slot, rep) packing lives in
FragmentNode::InverseWithLevel (src/layout/layout.cc:1133): rep
becomes a trailing ORDINARY input dim of extent ReplicateExtent()
(substituting the placeholder), thread a trailing output. The inverse
then maps (slot..., thread) -> (coords..., rep) with rep last;
loop_partition.cc consumes exactly this ordering and guards
replica-zero for stores. Mirror this packing when you need a fragment
as a plain multi-output layout.
- Layouts print via
DebugOutput() / Python repr:
Fragment((2,) -> (2,), replicate: 256, thread: _rep, index: (_i,), thread_range: I.Range(0, 256)). Structured fields are directly
accessible from Python: replicate_size, get_thread_size(),
forward_thread, forward_index, thread_range, get_input_shape().
Swizzles (tilelang/layout/swizzle.py, src/layout/swizzle_mode.*)
are XOR-based shared-memory layouts. They are NOT expressible as strided
layouts; the CuTe side models them as a separate Swizzle functor
(ComposedLayout), never as (shape, stride) modes.
CuTe layout algebra (src/layout/cute_layout.{h,cc}, tilelang/layout/cute.py)
A full CuTe implementation maintained for the MMA/TMA backends, exposed to
C++ (namespace tvm::tl::cute) and Python (tilelang.layout.cute):
- Layouts are
(shape, stride) IntTuple trees, COLUMN-major (first mode
fastest) — opposite of TileLang's row-major intuition. Multi-output
codomains use ScaledBasis strides (v@axis, E<i>).
- Algebra:
Coalesce, RightInverse, LeftInverse, Composition,
Complement, Cosize, LogicalDivide, Filter, Restrict,
Layout::WithShape, Parse/Print (exact CuTe spelling).
- Converters:
LayoutFromTileLang (single flat output; multi-output
tl::Layouts are serialized ROW-major, so outputs [thread, slot] yield
thread * slots + slot), LayoutFromTileLangHierarchical (per-axis
recovery onto basis axes), ComposedLayoutFromTileLang (swizzle
recovery). All use probe-then-prove: numerically probe strides at
one-hot points, then symbolically PROVE equivalence — a wrong recovery
cannot slip through; failure returns None/nullopt.
Failure conventions (must respect): only the three *FromTileLang
converters return Optional. Every other algebra op ICHECK-crashes on its
preconditions (composition divisibility, non-constant extents in the
probe, complement non-injectivity, restrict rank...). Wrap calls when the
input is untrusted. RightInverse is PARTIAL: it silently drops stride-0
and non-const-stride modes and inverts only the maximal contiguous chain —
check bijectivity via size(right_inverse(F)) == size(F).
Conversion caveats: the converters read only GetForwardIndex(); a
Fragment's forward_thread and replication are silently ignored unless
you pack them into a plain multi-output Layout first (see the canonical
packing above; substitute ReplicationPlaceholder with a trailing input
var — if it leaks into the probe, conversion gracefully returns None).
Layout inference (src/transform/layout_inference/)
tl.LayoutInference assigns a layout to every fragment buffer and
parallel loop nest. Three levels: strict (annotations, MMA-imposed) →
common (BFS propagation through shared buffers) → free (per
connected component, try every member as inference root, keep the
cheapest attempt). Results are stored as IR annotations: layout_map
(Buffer -> Layout, on the SBlock) and parallel_loop_layout (Fragment,
on the outermost parallel For); ParallelLoopLayoutValidator enforces the
annotation contract.
"Cheapest" is a pluggable policy (layout_cost_model.{h,cc}):
tl.layout_cost_model="io-aware" (opt-in): every
fragment<->global copy and global-touching parallel loop is charged
max(bandwidth bytes, issue bytes) under the attempt's layouts, scored
symbolically on the CuTe algebra (pack -> LayoutFromTileLang ->
RightInverse -> Composition; vector width read off coalesced modes;
segments counted at warp/step granularity). Registers are the tiebreak.
Statements outside the model are charged a conservative worst case — an
attempt must never profit from opacity.
tl.layout_cost_model="register-count" (default): register-slots-only
ordering.
- The scoring formulas are guarded by the Python parity check
maint/layout_inference/run.py --cute (symbolic scorer vs an
independent NumPy enumeration oracle); keep cute_model.py in lockstep
when changing layout_cost_model.cc.
Hardware geometry is parameterized: lane width via MaxVectorLoadBits
(src/transform/loop_vectorize.h, SHARED with the vectorizer so the
model's width beliefs match codegen), warp size from the target's
thread_warp_size, segment granularity 128B (see BindMemoryGeometry).
Downstream consumers
loop_partition.cc: partitions parallel loops per-thread via the
fragment inverse; emits replica-zero guards for replicated stores.
loop_vectorize.cc: plans vector widths (GetVectorizeSize,
IndicesCanVectorize); the cost model mirrors its judgment.
- TMA/MMA lowering:
src/cuda/op/tma_layout.cc and
producer_consumer_ws.cc recover shared-buffer swizzles via
ComposedLayoutFromTileLang; tcgen05/wgmma macro generators
(tilelang/cuda/intrinsics/macro/) round-trip TMEM layouts through
to_tilelang/from_tilelang_hierarchical.
Tools
maint/layout_inference/ — the layout verification harness (see its
README): golden layout snapshots per constructed case (run.py),
lowered vector-width anchors (--anchor), and symbolic-vs-oracle parity
(--cute, backed by cute_model.py + the numpy oracle.py). Extend it
with a new case whenever you change inference, the cost model, or the
converters; goldens are recorded then human-reviewed (--record).
- Diagnostics: the inference and cost-model passes log decisions via
DLOG (debug builds);
tl.layout_visualization_enable renders layouts.
- Python inspection: run
tl.transform.LayoutInference() on a module
and read the annotations (see maint/layout_inference/common.py for
the extraction idiom), or cute.Layout.from_tilelang(...) to see a
layout's (shape, stride) normal form.
File map
| area |
files |
| core types |
src/layout/layout.{h,cc}, tilelang/layout/{layout,fragment}.py |
| swizzle |
src/layout/swizzle_mode.*, tilelang/layout/swizzle*.py |
| CuTe algebra |
src/layout/cute_layout.{h,cc}, tilelang/layout/cute.py |
| inference |
src/transform/layout_inference/layout_inference.cc |
| cost model |
src/transform/layout_inference/layout_cost_model.{h,cc} |
| MMA layouts |
src/layout/gemm_layouts.cc, src/layout/tcgen05_layout.* |
| verification |
maint/layout_inference/ |
1---2name: tilelang-layout3description: Use when working with TileLang layouts — the Layout/Fragment classes, layout inference and its cost model, the CuTe layout algebra and TileLang-to-CuTe conversion, swizzles, or when debugging layout-related passes and their annotations.4metadata:5 source: https://skillmd.com/skills/tile-ai/tilelang-layout6---78# TileLang Layout System910## Core types (`src/layout/layout.h`, Python `tilelang/layout/`)1112**`tl::Layout`** maps logical coordinates to physical coordinates:13`input_size_` (logical shape) + `forward_index_` (one PrimExpr per output14axis). Expressions are written in process-wide placeholder Vars15`InputPlaceholder(i)` (printed `_i`, `_j`, ...). Key APIs:16`InputShape/OutputShape/GetForwardIndex/Forward(vars)/Inverse/17InverseWithLevel/Reshape/DetectInjective/DebugOutput`. `OutputShape()` is18DERIVED from the expressions' ranges by the Analyzer, not stored.1920**`tl::Fragment : Layout`** adds a thread dimension for register buffers:21`forward_thread_` (PrimExpr, may reference `ReplicationPlaceholder()`,22printed `_rep`) and `replicate_size_`. `ThreadExtent()` =23`max(forward_thread)+1` under bound domains — also derived, not stored.24`thread_range_` (via `BindThreadRange`) offsets the thread space for warp25specialization; the forward map itself stays in normalized `[0, T)`.26Replication semantics: each logical point is held by `replicate_size`27threads; `Replicate/DeReplicate/CondenseReplicateVar` manipulate the rep28axis.2930**Gotchas (verified, easy to trip on):**3132- `FragmentNode::GetForwardVars()` PREPENDS `ReplicationPlaceholder()`33 when `replicate_size > 1` (`src/layout/layout.cc:1103`). The input34 placeholders are always the TRAILING `InputDim()` entries. Python code35 that zips `get_forward_vars()` against the shape positionally is wrong36 for replicated fragments.37- The canonical (thread, slot, rep) packing lives in38 `FragmentNode::InverseWithLevel` (`src/layout/layout.cc:1133`): rep39 becomes a trailing ORDINARY input dim of extent `ReplicateExtent()`40 (substituting the placeholder), thread a trailing output. The inverse41 then maps `(slot..., thread) -> (coords..., rep)` with rep last;42 `loop_partition.cc` consumes exactly this ordering and guards43 replica-zero for stores. Mirror this packing when you need a fragment44 as a plain multi-output layout.45- Layouts print via `DebugOutput()` / Python `repr`:46 `Fragment((2,) -> (2,), replicate: 256, thread: _rep, index: (_i,),47 thread_range: I.Range(0, 256))`. Structured fields are directly48 accessible from Python: `replicate_size`, `get_thread_size()`,49 `forward_thread`, `forward_index`, `thread_range`, `get_input_shape()`.5051**Swizzles** (`tilelang/layout/swizzle.py`, `src/layout/swizzle_mode.*`)52are XOR-based shared-memory layouts. They are NOT expressible as strided53layouts; the CuTe side models them as a separate `Swizzle` functor54(`ComposedLayout`), never as (shape, stride) modes.5556## CuTe layout algebra (`src/layout/cute_layout.{h,cc}`, `tilelang/layout/cute.py`)5758A full CuTe implementation maintained for the MMA/TMA backends, exposed to59C++ (`namespace tvm::tl::cute`) and Python (`tilelang.layout.cute`):6061- Layouts are `(shape, stride)` IntTuple trees, **COLUMN-major (first mode62 fastest)** — opposite of TileLang's row-major intuition. Multi-output63 codomains use `ScaledBasis` strides (`v@axis`, `E<i>`).64- Algebra: `Coalesce`, `RightInverse`, `LeftInverse`, `Composition`,65 `Complement`, `Cosize`, `LogicalDivide`, `Filter`, `Restrict`,66 `Layout::WithShape`, `Parse`/`Print` (exact CuTe spelling).67- Converters: `LayoutFromTileLang` (single flat output; multi-output68 tl::Layouts are serialized ROW-major, so outputs `[thread, slot]` yield69 `thread * slots + slot`), `LayoutFromTileLangHierarchical` (per-axis70 recovery onto basis axes), `ComposedLayoutFromTileLang` (swizzle71 recovery). All use probe-then-prove: numerically probe strides at72 one-hot points, then symbolically PROVE equivalence — a wrong recovery73 cannot slip through; failure returns `None`/nullopt.7475**Failure conventions (must respect):** only the three `*FromTileLang`76converters return Optional. Every other algebra op ICHECK-crashes on its77preconditions (composition divisibility, non-constant extents in the78probe, complement non-injectivity, restrict rank...). Wrap calls when the79input is untrusted. `RightInverse` is PARTIAL: it silently drops stride-080and non-const-stride modes and inverts only the maximal contiguous chain —81check bijectivity via `size(right_inverse(F)) == size(F)`.8283**Conversion caveats:** the converters read only `GetForwardIndex()`; a84Fragment's `forward_thread` and replication are silently ignored unless85you pack them into a plain multi-output Layout first (see the canonical86packing above; substitute `ReplicationPlaceholder` with a trailing input87var — if it leaks into the probe, conversion gracefully returns None).8889## Layout inference (`src/transform/layout_inference/`)9091`tl.LayoutInference` assigns a layout to every fragment buffer and92parallel loop nest. Three levels: **strict** (annotations, MMA-imposed) →93**common** (BFS propagation through shared buffers) → **free** (per94connected component, try every member as inference root, keep the95cheapest attempt). Results are stored as IR annotations: `layout_map`96(Buffer -> Layout, on the SBlock) and `parallel_loop_layout` (Fragment,97on the outermost parallel For); `ParallelLoopLayoutValidator` enforces the98annotation contract.99100"Cheapest" is a pluggable policy (`layout_cost_model.{h,cc}`):101102- `tl.layout_cost_model="io-aware"` (opt-in): every103 fragment<->global copy and global-touching parallel loop is charged104 `max(bandwidth bytes, issue bytes)` under the attempt's layouts, scored105 symbolically on the CuTe algebra (pack -> `LayoutFromTileLang` ->106 `RightInverse` -> `Composition`; vector width read off coalesced modes;107 segments counted at warp/step granularity). Registers are the tiebreak.108 Statements outside the model are charged a conservative worst case — an109 attempt must never profit from opacity.110- `tl.layout_cost_model="register-count"` (default): register-slots-only111 ordering.112- The scoring formulas are guarded by the Python parity check113 `maint/layout_inference/run.py --cute` (symbolic scorer vs an114 independent NumPy enumeration oracle); keep `cute_model.py` in lockstep115 when changing `layout_cost_model.cc`.116117Hardware geometry is parameterized: lane width via `MaxVectorLoadBits`118(`src/transform/loop_vectorize.h`, SHARED with the vectorizer so the119model's width beliefs match codegen), warp size from the target's120`thread_warp_size`, segment granularity 128B (see `BindMemoryGeometry`).121122## Downstream consumers123124- `loop_partition.cc`: partitions parallel loops per-thread via the125 fragment inverse; emits replica-zero guards for replicated stores.126- `loop_vectorize.cc`: plans vector widths (`GetVectorizeSize`,127 `IndicesCanVectorize`); the cost model mirrors its judgment.128- TMA/MMA lowering: `src/cuda/op/tma_layout.cc` and129 `producer_consumer_ws.cc` recover shared-buffer swizzles via130 `ComposedLayoutFromTileLang`; tcgen05/wgmma macro generators131 (`tilelang/cuda/intrinsics/macro/`) round-trip TMEM layouts through132 `to_tilelang`/`from_tilelang_hierarchical`.133134## Tools135136- **`maint/layout_inference/`** — the layout verification harness (see its137 README): golden layout snapshots per constructed case (`run.py`),138 lowered vector-width anchors (`--anchor`), and symbolic-vs-oracle parity139 (`--cute`, backed by `cute_model.py` + the numpy `oracle.py`). Extend it140 with a new case whenever you change inference, the cost model, or the141 converters; goldens are recorded then human-reviewed (`--record`).142- **Diagnostics**: the inference and cost-model passes log decisions via143 DLOG (debug builds); `tl.layout_visualization_enable` renders layouts.144- **Python inspection**: run `tl.transform.LayoutInference()` on a module145 and read the annotations (see `maint/layout_inference/common.py` for146 the extraction idiom), or `cute.Layout.from_tilelang(...)` to see a147 layout's (shape, stride) normal form.148149## File map150151| area | files |152|---|---|153| core types | `src/layout/layout.{h,cc}`, `tilelang/layout/{layout,fragment}.py` |154| swizzle | `src/layout/swizzle_mode.*`, `tilelang/layout/swizzle*.py` |155| CuTe algebra | `src/layout/cute_layout.{h,cc}`, `tilelang/layout/cute.py` |156| inference | `src/transform/layout_inference/layout_inference.cc` |157| cost model | `src/transform/layout_inference/layout_cost_model.{h,cc}` |158| MMA layouts | `src/layout/gemm_layouts.cc`, `src/layout/tcgen05_layout.*` |159| verification | `maint/layout_inference/` |