# Lc Optimize

> Optimize LuisaCompute DSL kernels using warp/wave primitives, shared-memory aggregation, and block-level collectives. Use when kernels bottleneck on atomics, reductions, or inter-thread communication.

- Skill: `luisagroup/lc-optimize` (Agent Skill)
- Install (CLI): `npx skillmds@latest add luisagroup/lc-optimize`
- Raw SKILL.md: https://api.skillmd.com/api/skills/luisagroup/lc-optimize/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: luisagroup (https://skillmd.com/u/luisagroup)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/luisagroup/lc-optimize

---


# LuisaCompute DSL Kernel Optimization Guide

> **Zero-initialization note:** All temporary local variables in DSL kernels — scalars, vectors, matrices, structs, and arrays (excluding shared arrays `Shared<T>`) — are created with a zero value automatically. There is no need to manually set them to zero before use. This applies to variables declared with `Var<T>`, `auto`, or type-inferred syntax inside a kernel or callable body.

## 1. Available Warp/Wave Primitives

LuisaCompute exposes the following warp-level (subgroup) intrinsics via `luisa/dsl/builtin.h`. All operate on *active lanes within the current warp*.

### 1.1 Query / Metadata

| DSL Call | Returns | Description |
|---|---|---|
| `warp_lane_count()` | `UInt` | Total lanes in the warp (e.g. 32 or 64). |
| `warp_lane_id()` | `UInt` | Current lane index `[0, warp_lane_count())`. |
| `warp_is_first_active_lane()` | `Bool` | True if this lane is the first active lane in the warp. |
| `warp_first_active_lane()` | `UInt` | Lane index of the first active lane. |
| `device.compute_warp_size()` | `uint` | Host-side query for backend's native warp size. |

### 1.2 Active-Lane Reductions (All-Reduce)

Each lane receives the same reduced value.

| DSL Call | Signature | Semantics |
|---|---|---|
| `warp_active_sum(v)` | `T -> T` | Sum across active lanes. |
| `warp_active_product(v)` | `T -> T` | Product across active lanes. |
| `warp_active_min(v)` | `T -> T` | Minimum across active lanes. |
| `warp_active_max(v)` | `T -> T` | Maximum across active lanes. |
| `warp_active_all(v)` | `Bool -> Bool` | Logical AND across active lanes. |
| `warp_active_any(v)` | `Bool -> Bool` | Logical OR across active lanes. |
| `warp_active_all_equal(v)` | `T -> Bool` (or `Vec<Bool,N>`) | True if all active lanes have the same value. |
| `warp_active_bit_and(v)` | `Int -> Int` | Bitwise AND across active lanes. |
| `warp_active_bit_or(v)` | `Int -> Int` | Bitwise OR across active lanes. |
| `warp_active_bit_xor(v)` | `Int -> Int` | Bitwise XOR across active lanes. |
| `warp_active_count_bits(v)` | `Bool -> UInt` | Population count of true predicates. |
| `warp_active_bit_mask(v)` | `Bool -> UInt4` | 128-bit ballot mask of true predicates. |

All accept scalar or vector types (except bit ops which require integral types).

### 1.3 Prefix (Exclusive Scan)

Each lane receives the exclusive prefix of all *preceding* active lanes. Lane 0 receives identity (0 for sum, 1 for product, 0u for count_bits).

| DSL Call | Signature | Description |
|---|---|---|
| `warp_prefix_sum(v)` | `T -> T` | Exclusive prefix sum (T: arithmetic). |
| `warp_prefix_product(v)` | `T -> T` | Exclusive prefix product (T: arithmetic). |
| `warp_prefix_count_bits(v)` | `Bool -> UInt` | Exclusive prefix popcount of true predicates. |

### 1.4 Lane Communication (Shuffle)

| DSL Call | Signature | Description |
|---|---|---|
| `warp_read_lane(v, lane_idx)` | `(T, UInt) -> T` | Read `v` from lane `lane_idx`. T can be scalar, vector, or matrix. |
| `warp_read_first_active_lane(v)` | `T -> T` | Read `v` from the first active lane. |

### 1.5 Configuration

| DSL Call | Description |
|---|---|
| `set_warp_size(uint8_t)` | Must be a power-of-two in [8,128]. Call *inside* kernel lambda before compilation. |
| `sync_block()` | Full block barrier. All threads in the block must reach it. |

**Critical rule:** Warp operations only communicate within the *same warp*. No `sync_block()` is needed for warp collectives — they are guaranteed to complete within the warp without barriers.

---

## 2. Usage Patterns (Project Analysis)

### 2.1 Warp-Level Matrix Multiplication

Pattern: each warp computes one output tile via `warp_active_sum` reduction over the K dimension.

```cpp
auto warp_size = device.compute_warp_size();
Kernel2D mat_mul = [&](BufferFloat lhs, BufferFloat rhs, BufferFloat result, UInt lhs_row_size) {
    set_block_size(128, 1, 1);
    set_warp_size(warp_size);

    UInt lhs_y = dispatch_id().x / warp_size;
    UInt rhs_x = dispatch_id().y;
    UInt warp_lane = warp_lane_id();

    UInt tile_count = (lhs_row_size + warp_size - 1) / warp_size;
    Float acc = 0.f;

    for (auto t : dynamic_range(tile_count)) {
        UInt lhs_x = t * warp_size + warp_lane;
        Float local_v;
        $if (lhs_x < lhs_row_size) {
            local_v = lhs.read(lhs_y * lhs_row_size + lhs_x)
                    * rhs.read(rhs_x * lhs_row_size + lhs_x);
        } $else {
            local_v = 0.f;
        };
        acc += warp_active_sum(local_v);   // all-reduce sum within warp
    }

    $if (warp_lane == 0) {
        result.write(rhs_x * /*M*/ ...  + lhs_y, acc);
    };
};
```

**Key insight:** Warp-active reductions eliminate the need for shared memory entirely. All lanes in a warp already execute in lockstep, so `warp_active_sum` is a single hardware instruction on most GPUs.

### 2.2 Butterfly Reduction via `warp_read_lane`

For finding the maximum across a logical group smaller than the warp, use pairwise `warp_read_lane` with XOR lane masks (butterfly / tree-reduction pattern):

```cpp
// 8-lane max reduction within a group of 8
Float m = input;
m = max(m, warp_read_lane(m, lane ^ 4u));  // distance 4
m = max(m, warp_read_lane(m, lane ^ 2u));  // distance 2
m = max(m, warp_read_lane(m, lane ^ 1u));  // distance 1
// Now lane 0..7 all have the max of lanes 0..7 (modulo diverged lanes)
```

This is used for the softmax normalization constant. Each logical group of 8 lanes computes its own max independently, *without* a barrier.

### 2.3 Grouped Prefix + Inter-Group Read

When a warp contains multiple independent logical groups, compute the inclusive prefix sum per group, then use `warp_read_lane` to fetch the last element of the previous group:

```cpp
constexpr uint kWarpSize = 32u;
constexpr uint kGroupLanes = 8u;
constexpr uint kGroupsPerWarp = kWarpSize / kGroupLanes;  // 4

auto lane = warp_lane_id();
auto group_id = lane / kGroupLanes;      // which group (0..3)
auto group_lane = lane % kGroupLanes;    // position within group (0..7)

auto prefix = warp_prefix_sum(value);    // exclusive prefix across whole warp
auto inclusive = prefix + value;

// Last lane of this group
auto last_lane = group_id * kGroupLanes + (kGroupLanes - 1u);
auto incl_last = warp_read_lane(inclusive, last_lane);

// Last lane of previous group (or 0 for group 0)
auto prev_last = ite(group_id == 0u, 0u, last_lane - kGroupLanes);
auto prev_incl = warp_read_lane(inclusive, prev_last);

auto group_sum = incl_last - ite(group_id == 0u, make_float2(0.f), prev_incl);
```

This avoids separate `warp_prefix_sum` calls per group and instead uses a single warp-wide prefix plus lane reads to extract group boundaries.

### 2.4 Warp-Polling Decoupled Look-Back

For inter-block scan, tiles publish their status and other tiles poll via warp collectives:

```cpp
// Poll across warp: any lane sees INVALID?
$while (warp_active_any(status == SCAN_TILE_INVALID)) {
    delay();
    status = tile_status.volatile_read(predecessor_idx);
};

// All lanes agree predecessor is inclusive?
$while (warp_active_all(predecessor_status != SCAN_TILE_INCLUSIVE)) {
    predecessor_idx -= 32;
    // poll next window...
    exclusive = scan_op(window_aggregate, exclusive);
};
```

Also uses `warp_active_bit_mask` for segmented reductions within warps:

```cpp
UInt warp_flags = warp_active_bit_mask(flag == 1u).x;
warp_flags >>= 1;  // for HEAD_SEGMENT mode
warp_flags &= get_lane_mask_ge();  // mask of lanes with id >= mine
warp_flags |= 1u << (LOGIC_WARP_SIZE - 1u);  // sentinel
UInt last_lane = ctz(warp_flags);  // first set bit = end of my segment
```

### 2.5 Shuffle-Down for Warp Reduction

A software implementation of warp reduce using `warp_read_lane` with increasing offsets:

```cpp
Var<T> result = input;
UInt offset = 1u;
$while (offset < warp_lane_count()) {
    Var<T> temp = warp_read_lane(result, lane_id + offset, valid_item);
    $if (lane_id + offset <= valid_item) {
        result = reduce_op(result, temp);
    };
    offset <<= 1;
};
```

This is a fallback pattern; prefer `warp_active_sum` / `warp_active_min` / `warp_active_max` when the operation matches the built-in.

### 2.6 Quantized Matmul with Warp

Warp-level GEMM where each warp computes one output tile. Threads cooperatively load quantized weights via `warp_read_lane` to assemble dequantized values, then accumulate with `warp_active_sum`:

```cpp
auto warp_lane = warp_lane_id();
UInt tile_count = (K + warp_size - 1) / warp_size;
for (auto t : dynamic_range(tile_count)) {
    UInt tile_begin = t * warp_size;
    UInt tile_size = min(warp_size, K - tile_begin);

    // Each lane loads one quantized word, then shares via warp_read_lane
    UInt rel_byte = warp_lane * kElementByteSize;
    UInt word = warp_read_lane(warp_word, rel_byte / 4u);
    // ... dequantize and multiply ...

    acc += warp_active_sum(local_v);
}
```

---

## 3. Optimization Transformations

### 3.1 Shared-Memory Atomic → Warp Collective

**Before:** Block-level atomic on shared memory.
```cpp
Shared<int> shared{1u};
shared[0u] = 0;
sync_block();
shared.atomic(0u).fetch_add(1);  // contended within block
sync_block();
$if (thread_x() == 0u) {
    global_counter.atomic(0u).fetch_add(shared.read(0u));
};
```

**After:** Use `warp_active_sum` per warp, then one lane writes.
```cpp
// Each warp computes its own partial sum
Int warp_partial = warp_active_sum(1);
// First lane of each warp writes to shared
$if (warp_is_first_active_lane()) {
    shared.atomic(warp_lane_id() / warp_size).fetch_add(warp_partial);
};
sync_block();
// One thread (e.g. thread 0) combines across warps
$if (thread_x() == 0u) {
    Int block_total = 0;
    for (auto w : range(num_warps_per_block)) {
        block_total += shared.read(w);
    };
    global_counter.atomic(0u).fetch_add(block_total);
};
```

### 3.2 Shared-Memory Reduction → Warp Reduction

**Before:** Entire block reduces into shared memory.
```cpp
Shared<float> smem{block_size};
smem[tid] = value;
sync_block();
for (uint stride = block_size / 2; stride > 0; stride >>= 1) {
    $if (tid < stride) {
        smem[tid] += smem[tid + stride];
    };
    sync_block();
};
```

**After:** Warp-level reduction + cross-warp shared reduction.
```cpp
// Phase 1: warp-level reduction (no shared memory, no barrier)
Float warp_sum = warp_active_sum(value);

// Phase 2: cross-warp reduction in shared memory (much smaller)
$if (warp_is_first_active_lane()) {
    Shared<float> warp_results{num_warps_per_block};
    warp_results[warp_id()] = warp_sum;
};
sync_block();
// Only num_warps_per_block threads participate in phase 2
$if (thread_x() < num_warps_per_block) {
    Float v = warp_results[thread_x()];
    Float warp_partial = warp_active_sum(v);  // second warp reduce
    $if (warp_is_first_active_lane()) {
        result = warp_partial;
    };
};
```

### 3.3 Pairwise Max/Min → Built-in Warp Max/Min

**Before:** Butterfly pattern with `warp_read_lane`.
```cpp
Float m = value;
m = max(m, warp_read_lane(m, lane ^ 4u));
m = max(m, warp_read_lane(m, lane ^ 2u));
m = max(m, warp_read_lane(m, lane ^ 1u));
```

**After:** Single `warp_active_max` when reduction spans the *whole* warp.
```cpp
Float m = warp_active_max(value);
```

**When to keep butterfly:** Only when reducing over a logical group *smaller* than the warp (e.g. 8-lane groups inside a 32-lane warp). In that case `warp_active_max` would reduce over all 32 lanes, which is incorrect.

### 3.4 Sequential Lane Reads → `warp_prefix_sum`

**Before:** Manually accumulating values from lower lanes via a loop of `warp_read_lane`.
```cpp
Float prefix = 0.f;
for (uint i = 0; i < warp_lane_id(); i++) {
    prefix += warp_read_lane(value, i);
};
```

**After:** Single `warp_prefix_sum` call.
```cpp
Float prefix = warp_prefix_sum(value);
```

### 3.5 Conditional Participation

When only a subset of lanes should participate in a warp collective, wrap the call in a conditional. Lanes that *don't* execute the call are excluded from the reduction/scan:

```cpp
$if (thread_x() % 2u == 0u) {
    // Only even-index threads contribute
    auto result = warp_prefix_sum(make_half4(.5_h));
    device_log("{} -> {}", dispatch_x(), result);
};
// Odd threads don't participate; they receive no result.
```

This pattern is commonly used for partial-warp scans where only a subset of lanes needs results.

### 3.6 Ballot + Count Bits for Control Flow

Use `warp_active_bit_mask` to build a lane mask, then `ctz` / `popcount` to locate lanes or count participants:

```cpp
UInt4 mask = warp_active_bit_mask(condition);
UInt flag_mask = mask.x;  // first 32 lanes
UInt first_true = ctz(flag_mask);
UInt num_true = popcount(flag_mask);
```

---

## 4. Shared Array (Workgroup Memory) Optimization

Warp collectives (section 3) only communicate *within one warp*. When cooperation must span the **whole thread block** (multiple warps), or you need persistent per-block scratch, arbitrary cross-thread indexing, or block-local privatization of a global atomic, use a **shared array** (`Shared<T>` / `$shared<T>`). Shared memory is on-chip and orders of magnitude faster than global memory, so staging data there once and reusing it, or aggregating locally before touching global memory, is a core optimization.

### 4.1 API (`include/luisa/dsl/shared.h`, `include/luisa/dsl/sugar.h:105`)

| DSL | Description |
|---|---|
| `Shared<T> s{n}` / `$shared<T> s{n}` | Allocate `n` elements of `T` in workgroup memory. Must be constructed **inside** the kernel/callable body (uses `FunctionBuilder::current()`). |
| `s[i]` | Reference access (read or write); `i` must be an integral expr. |
| `s.read(i)` / `s.write(i, v)` | Explicit read / write helpers (alias for `s[i]`). |
| `s.atomic(i).fetch_add(v)` / `.compare_exchange(e, v)` / ... | Atomic ops on a shared slot. Available for scalar/vector element types (disabled for custom structs). |
| `s.size()` | Element count. |
| `new Shared<T>{n}` | Heap-allocate so helper classes can *own* shared scratch (`Shared<T>` is move-only, non-copyable). Lifetime is tied to the enclosing kernel's function builder. See `WarpReduce` in `test_decoupled_look_back.cpp`. |

Always `set_block_size(...)` and size the array to the block (`Shared<T> s{block_size}`). Use `sync_block()` to make writes visible across warps.

### 4.2 When to prefer shared memory over warp collectives

| Situation | Use |
|---|---|
| Reduction/scan fits in a single warp | Warp collective (section 3) — no barrier, single instruction. |
| Reduction spans a whole block (block_size > warp_size) | Two-level: warp collective → shared → block (section 4.5), or full shared-memory tree reduction (4.4). |
| Many threads append to one global counter/queue | Block-local privatization in shared, then **one** global atomic per block (4.3). |
| Global data reused by many threads in a block | Stage global → shared once, `sync_block()`, then reuse (4.6). |
| Arbitrary cross-thread indexing (not just lane shuffles) | Shared array indexed by `thread_id()`. |

### 4.3 Block-Local Atomic Privatization → One Global Atomic

The biggest shared-memory win: replace *up to block_size* contended **global** atomics with per-thread **shared** atomics plus a **single** global atomic per block. Pattern from `test_atomic_queue.cpp` (`push_if`) and `test_shared_memory.cpp` (`AtomicQueue::push`):

```cpp
// Append `value` to a global queue when `pred` holds, minimizing global contention.
Shared<uint> index{1};
$if (thread_x() == 0u) { index.write(0u, 0u); };   // init block counter
sync_block();

auto local_index = def(0u);
$if (pred) { local_index = index.atomic(0).fetch_add(1u); };  // cheap SHARED atomic
sync_block();

$if (thread_x() == 0u) {                            // ONE global atomic for the whole block
    auto local_count   = index.read(0u);
    auto global_offset = _counter->atomic(0u).fetch_add(local_count);
    index.write(0u, global_offset);                 // reuse slot to broadcast the base
};
sync_block();

$if (pred) {                                        // scatter to reserved, contiguous range
    auto global_index = index.read(0u) + local_index;
    _buffer->write(global_index, value);
};
```

**Insight:** Global-atomic traffic drops from O(active threads) to O(1) per block. Contention moves from device-wide global memory to fast on-chip shared memory. This is the standard stream-compaction / queue-append optimization.

### 4.4 Block-Wide Tree Reduction in Shared Memory

When the reduction spans the whole block, stage each thread's value in shared memory and reduce pairwise with a halving loop. Pattern from `test_softmax.cpp` (block sum for softmax) and `test_complex_kernel.cpp`:

```cpp
set_block_size(block_size, 1, 1);          // power of two
Shared<float> shared_arr(block_size);       // one slot per thread
auto tid = thread_id().x;
shared_arr[tid] = value;                     // stage per-thread value

UInt half = block_size / 2u;
sync_block();
$while (half > 0u) {
    $if (tid < half) {                       // compute into a register FIRST
        value = shared_arr[tid * 2] + shared_arr[tid * 2 + 1];
    };
    sync_block();                            // barrier between read and write-back
    $if (tid < half) {
        shared_arr[tid] = value;             // write reduced value back
    };
    half /= 2u;
    sync_block();                            // barrier before next iteration's reads
};
$if (tid == 0u) { output.write(block_id().x, shared_arr[0]); };  // thread 0 emits block result
```

**Why two `sync_block()` per step:** reducing into a local `value` register and only writing back after a barrier avoids the read-after-write / write-after-read hazard where one thread overwrites a slot another thread is still reading. Prefer this whole-block form only when `block_size > warp_size`; inside a single warp, `warp_active_sum` (section 3.2) is faster and barrier-free.

### 4.5 Two-Level Reduction: Warp Collective → Shared → Block

Combine both tools: reduce within each warp with a warp collective (no barrier), write one partial per warp to a *small* shared array, then reduce those partials. This minimizes both shared traffic and barriers vs. a full block tree reduction (see also sections 3.1/3.2):

```cpp
UInt warp_id = thread_x() / warp_size;            // which warp in the block (warp_size = device.compute_warp_size())
Float warp_sum = warp_active_sum(value);          // phase 1: intra-warp, no barrier
Shared<float> warp_results{num_warps_per_block};
$if (warp_is_first_active_lane()) {
    warp_results[warp_id] = warp_sum;             // one write per warp
};
sync_block();
$if (thread_x() < num_warps_per_block) {          // phase 2: reduce the few partials
    Float block_sum = warp_active_sum(warp_results[thread_x()]);
    $if (warp_is_first_active_lane()) { /* thread 0 has the block total */ };
};
```

### 4.6 Shared as Staging / Scratch for Reuse & Exchange

Load global data into shared once, then reuse it many times or exchange it between threads, avoiding repeated global reads. Patterns from `test_shared_mem.cpp`, `test_async_copy.cpp`, and hierarchical mip reduction in `test_mipmap.cpp`:

```cpp
set_block_size(N, 1u, 1u);
Shared<uint> s_src{N};
auto tid = thread_x();
s_src[tid] = src_buf.read(dispatch_x());     // global -> shared, once
sync_block();                                 // publish to the whole block
// ... now reuse s_src[...] / read neighbors' values without touching global memory ...
```

`test_async_copy.cpp` fills a shared staging buffer with `async_copy(...)` (thread 0 issues the copy, then `sync_block()` before consumers read). `test_mipmap.cpp` writes 2×2 block averages into `Shared<float3>` and reduces level-by-level with a `sync_block()` between levels.

### 4.7 Correctness & Performance Rules

1. **Construct inside the kernel body.** `Shared<T>` needs `FunctionBuilder::current()`; declaring it outside a kernel/callable is invalid.
2. **Barrier discipline.** A `sync_block()` is required (a) after initializing/filling shared before other threads read, and (b) between the read and write-back phases of each reduction step. Unlike warp collectives, **shared memory is NOT self-synchronizing across warps**.
3. **Size to the block.** Match the array length to `set_block_size(...)`; use a power-of-two block for the halving tree reduction, and pad out-of-range lanes with the reduction identity (e.g. `0.f` for sum) — see the `$if (id < size) {...} $else { value = 0.f; }` guards in `test_softmax.cpp`.
4. **Register-then-write.** In tree reductions, compute into a `Var`/register and write back only after a barrier to avoid RAW/WAR hazards.
5. **Move-only ownership.** `Shared<T>` cannot be copied; store `Shared<T> *` (via `new`) when a helper class must hold shared scratch.
6. **Prefer warp collectives when they suffice.** Shared memory costs a barrier and on-chip capacity; only reach for it when cooperation exceeds one warp or needs privatization/staging/arbitrary indexing.

---

## 5. Hardware Mapping

| GPU Backend | Warp/Lane Terminology | Native Width |
|---|---|---|
| CUDA | Warp (32 lanes) | 32 |
| HIP | Wavefront (32/64 lanes) | 32 or 64 |
| Vulkan | Subgroup | Varies (usually 32 or 64) |
| DirectX | Wave | 32 or 64 |
| Metal | SIMD group | 32 |

Always query `device.compute_warp_size()` on the host and `warp_lane_count()` on the device rather than hardcoding 32.

---

## 6. Rules of Thumb

1. **Prefer warp collectives over shared memory.** `warp_active_sum`, `warp_active_max`, `warp_prefix_sum` compile to single hardware instructions (e.g. `__shfl_xor_sync` on CUDA, `OpGroupNonUniformFAdd` on SPIR-V). No barrier needed.

2. **Set warp size explicitly** when using warp collectives: `set_warp_size(device.compute_warp_size())` inside the kernel lambda.

3. **Don't mix warp and block assumptions.** `warp_active_sum` only reduces within the current warp. If you have multiple warps per block, use a two-level reduction (warp → shared → block).

4. **`sync_block()` is NOT needed between warp collectives** within the same warp. Warp ops are guaranteed complete for the calling lane immediately.

5. **Divergence matters.** Lanes that don't execute the warp collective call are excluded. Use this for conditional participation (section 3.5).

6. **`warp_prefix_sum` is exclusive** (not inclusive). Lane 0 always gets 0 (for sum) or 1 (for product).

7. **Vector types work.** All warp collectives accept `float2`, `float3`, `float4`, `int2`, etc. The operation applies component-wise.

8. **Logic warp size.** You can logically group lanes (e.g. 4 groups of 8 within a 32-lane warp) using `lane % kGroupLanes` and `lane / kGroupLanes` arithmetic. Use `warp_read_lane` to communicate across groups.

9. **Use shared memory for block-wide cooperation** (section 4). When cooperation exceeds one warp, or you need privatization/staging/arbitrary cross-thread indexing, `Shared<T>` beats warp collectives. Always `set_block_size` and size the array to the block.

10. **Privatize global atomics into shared memory** (section 4.3). Aggregate per-thread contributions with cheap shared atomics, then issue **one** global atomic per block. This is the key stream-compaction / queue-append win.

11. **Shared memory needs `sync_block()`; warp collectives do not.** Barrier after filling shared and between the read/write-back phases of a tree reduction (section 4.4). Reduce into a register first, then write back after the barrier to avoid RAW/WAR hazards.

---

## 7. Host-Side Command Batching with CommandList

GPU kernel optimization (sections 1–6) focuses on what happens *inside* a single kernel dispatch. Equally important is how you *submit* work from the host: every `stream << command` call adds driver overhead. When a per-frame or per-iteration hot loop issues many small stream submissions (upload, dispatch A, dispatch B, download, ...), the accumulated driver cost can become a bottleneck, especially on D3D12 and Vulkan where command submission is not free.

### 7.1 The Pattern

`CommandList` lets you batch multiple commands into a single submission. Commands are recorded into a `CommandList` object, then committed to the stream in one shot:

```cpp
CommandList cmdlist = CommandList::create();
cmdlist << upload_command
        << dispatch_a
        << dispatch_b
        << download_command;
stream << cmdlist.commit() << synchronize();
```

All commands execute in FIFO order on the GPU, exactly as if they were submitted individually — but with a single driver round-trip instead of many.

### 7.2 When to Use

- **Hot loops**: Rendering loops, training iterations, or per-frame update loops that submit several commands each iteration.
- **Dependent pipeline stages**: Commands with producer-consumer relationships (e.g., kernel A writes a buffer, kernel B reads it) that can be submitted together because GPU ordering guarantees correct sequencing.
- **Upload → compute → download chains**: Batched uploads followed by multiple kernels then final downloads, all in one commit.

### 7.3 When NOT to Use

- **Interactive latency-sensitive paths**: If a command produces results that need immediate host feedback (e.g., debug readbacks), avoid delaying it behind unrelated work.
- **Cross-stream synchronization**: Commands in different streams (COMPUTE vs GRAPHICS) must use events for ordering; a single `CommandList` cannot span multiple streams.
- **Very long command sequences**: Extremely large command lists may starve the GPU if they take too long to record; split into chunks if recording itself becomes a bottleneck.

### 7.4 General Recipe

1. **Identify the hot loop** — look for repeated `stream <<` statements inside a loop or per-frame function.
2. **Group dependent commands** — all commands that form an in-order GPU pipeline (upload → kernel A → kernel B → download) belong in the same `CommandList`.
3. **Create and fill** — call `CommandList::create()` once at the start of the group, then append commands with `<<`.
4. **Commit once** — `stream << cmdlist.commit()` submits the batch; follow with a single `synchronize()` if host-readback is needed.
5. **Verify correctness** — ensure the sequence of operations inside the CommandList matches the dependency order (commands execute in FIFO order on the GPU).

### 7.5 Performance Impact

Batching N separate `stream << cmd` submissions into one `CommandList` reduces:
- **Driver submission overhead**: Each stream submission incurs a kernel transition / command-queue flush cost. With CommandList, that cost is paid once per batch.
- **Host-device synchronization points**: A single `commit() + synchronize()` replaces N pairs of `stream << ... << synchronize()`.

In practice, replacing 5+ stream submissions per iteration with a single `CommandList::create()` → `commit()` can yield measurable wall-clock speedups in offline rendering or training-data export loops, where the CPU-side submission overhead is a meaningful fraction of the iteration time.

### 7.6 Comparison with Other Optimizations

| Optimization | Scope | Impact |
|---|---|---|
| Warp collectives (section 3) | GPU kernel — replaces shared memory and atomics | Reduces latency/contention within a warp |
| Shared memory privatization (section 4.3) | GPU kernel — aggregates block atomics | Reduces global atomic contention from O(block_size) to O(1) per block |
| **CommandList batching** (this section) | **Host submission — batches stream commands** | **Reduces driver overhead from O(N) to O(1) per iteration** |

CommandList batching is orthogonal to kernel-level optimizations. Apply both: optimize the kernel with warp/shared-memory techniques, then batch the host submissions for maximum throughput.

### 7.7 Key Rules

1. **One CommandList, one commit, one sync.** Create a single `CommandList` for a group of dependent commands, commit it once, and synchronize once rather than submitting each command separately.
2. **FIFO ordering preserved.** Commands execute in record order — no need for explicit barriers between kernel dispatches and buffer copies inside the same CommandList (GPU pipeline dependencies are handled automatically).
3. **Don't reuse a committed CommandList.** After `commit()` the list is consumed; create a fresh one for the next batch.
4. **Prefer CommandList over chaining on `stream <<`.** Batched submission is more efficient than long chains of `stream << a << b << c << synchronize()` because it reduces internal queue flushes.
5. **Combine with kernel optimization.** Host-side batching and kernel-level warp/shared-memory optimization are complementary — use both.

### 7.8 Async Callbacks — Replacing `synchronize()` with Non-Blocking Completion

`CommandList` provides two callback hooks that decouple host work from GPU execution:

```cpp
// Runs AFTER all GPU commands in this CommandList finish.
cmdlist.add_callback([](auto &&... captured) noexcept {
    // GPU work is done — safe to read back buffers, write files, etc.
});

// Runs when the CommandList is committed/destructed, BEFORE GPU starts.
cmdlist.add_dtor_callback([](auto &&... captured) noexcept {
    // Host-side cleanup of resources no longer needed.
});
```

#### 7.8.1 Understanding the Two Callbacks

| Callback | When it fires | GPU status | Typical use |
|---|---|---|---|
| `add_dtor_callback` | At `commit()` (or when `Commit` object is destroyed) | **Not started yet** | Release temporary host buffers, close files, or free staging memory that was only needed to construct the commands. |
| `add_callback` | After all GPU commands in the list have completed | **Done** | Read back downloaded buffers, write output files, signal host work queues, or launch dependent host tasks. |

**Critical difference:** `add_dtor_callback` runs *before* GPU execution begins — it is not a completion callback. Only `add_callback` guarantees GPU work is finished.

#### 7.8.2 Avoiding `synchronize()` Stalls

Without callbacks, host→GPU data exchange typically looks like:

```cpp
// Blocking pattern: host stalls until GPU finishes.
stream << cmdlist.commit() << synchronize();  // host waits here
process_results(host_buffer);                  // then processes
```

`add_callback` lets you flip this into a non-blocking, continuation-passing style:

```cpp
// Non-blocking pattern: callback processes results when GPU is done.
cmdlist.add_callback([host_buffer = std::move(host_buf)]() noexcept {
    process_results(host_buffer);  // runs after GPU finishes
});
stream << cmdlist.commit();  // host returns immediately, no stall
// Host can begin preparing the next frame/iteration NOW...
```

This is most impactful when:
- The host has independent work (e.g., preparing the next config, loading assets, updating UI).
- The GPU work is long enough that blocking would waste host cycles.
- You process results per-iteration and can pipeline iterations (iteration N's callback runs while iteration N+1's GPU work is already in flight).

#### 7.8.3 The Pipelined Iteration Pattern

The classic pattern for hiding latency: overlap GPU execution of iteration N+1 with host processing of iteration N's results.

```cpp
// Host-side buffers must outlive the GPU work.
// Use double-buffering or shared ownership (e.g., shared_ptr).
for (int i = 0; i < num_iterations; i++) {
    auto host_buf = std::make_shared<std::vector<float>>(size);
    CommandList cmdlist = CommandList::create();

    // Record GPU commands (upload, dispatch, download into host_buf)
    cmdlist << upload << kernel.dispatch(...) << download(host_buf->data());

    // Install the completion callback — captures host_buf by shared_ptr
    cmdlist.add_callback([host_buf, i]() noexcept {
        // GPU done: safely read host_buf, write to disk, etc.
        save_result(*host_buf, i);
    });

    // Submit without blocking — host returns immediately
    stream << cmdlist.commit();
    // Host can prepare iteration i+1's work right away...
}
// Final sync: wait for the very last iteration to finish.
stream << synchronize();
```

With this pattern:
- **No `synchronize()` per iteration** — only one final sync at the very end.
- **CPU and GPU overlap** — iteration N's result processing runs concurrently with iteration N+1's GPU execution.
- **Throughput improves** by the cost of one `synchronize()` stall × (N−1) iterations.

#### 7.8.4 Capturing Resources for Callbacks

Lambdas passed to `add_callback` / `add_dtor_callback` must own their captured resources because the callback outlives the `CommandList` object. Use:

```cpp
// ✅ Shared ownership (recommended for buffers)
auto data = std::make_shared<std::vector<float>>(size);
cmdlist.add_callback([data]() noexcept { /* safe */ });

// ✅ Move semantics for unique resources
auto data = std::make_unique<std::vector<float>>(size);
cmdlist.add_callback([data = std::move(data)]() noexcept { /* safe */ });

// ❌ Capturing raw pointers or references to stack/local variables is UAF.
float *raw = ...;
cmdlist.add_callback([raw]() noexcept { /* DANGER: raw may be dangling */ });
```

`add_dtor_callback` has the same ownership rules, despite running earlier — it still fires after `commit()` returns, so stack variables captured by reference would be invalid.

#### 7.8.5 When to Use Which

| Situation | Use |
|---|---|
| Read back GPU results and save/process them | `add_callback` |
| Free host staging buffers after upload | `add_dtor_callback` |
| Close files or decrement refcounts after submission | `add_dtor_callback` |
| Signal a host work queue that GPU output is ready | `add_callback` |
| Launch the next iteration's host prep work | Just place after `commit()` on the host (no callback needed) |

#### 7.8.6 Key Rules

1. **`add_callback` fires after GPU completion** — it is the non-blocking replacement for `synchronize()`.
2. **`add_dtor_callback` fires before GPU starts** — use only for host-side cleanup, never for reading back GPU results.
3. **Always capture by value** (shared_ptr, unique_ptr, or copy). Raw pointers and references to stack variables are dangling by the time the callback runs.
4. **One final `synchronize()` is still needed** at the end of a pipeline to ensure the last iteration's callbacks have fired before the program exits.
5. **Callbacks execute on an internal worker thread** — they should not throw, block on the GPU, or perform GPU API calls on the same stream.