# Opt Buffer Object Reuse

> Purpose

- Skill: `xilinx/opt-buffer-object-reuse` (Agent Skill)
- Install (CLI): `npx skillmds@latest add xilinx/opt-buffer-object-reuse`
- Raw SKILL.md: https://api.skillmd.com/api/skills/xilinx/opt-buffer-object-reuse/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: xilinx (https://skillmd.com/u/xilinx)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/xilinx/opt-buffer-object-reuse

---


## Purpose

NPU kernels re-allocate and re-upload BufferObjects (BOs) on every call by
default. For an N-layer transformer that re-runs the same kernels per layer
(prefill) and per token (decode), this is pure redundant host↔NPU traffic.
This skill removes it. It is the same optimization the reference ablation
isolated as cells A→B (weight BOs) and B→C (intermediate BOs); together they
were a multi-second prefill saving and the dominant decode host-side cost.

Two mechanics, one class:

- **B1 — per-layer weight BOs** (`static_input_indices`): allocate each
  layer's weight BOs once during setup, write them once, and pass
  `static_input_indices=[<weight slots>]` on every `cache.load_and_run()` so
  the runtime skips re-writing them.
- **B2 — intermediate BOs** (`intermediate_indices`): for buffers the kernel
  fully overwrites (its own outputs / scratch), pass
  `intermediate_indices=[<output slots>]` so the host does not write them
  before the call.

## Success criteria

Applying this skill is "successful" when ALL hold:

1. Output cosine ≥ 0.99 vs the pre-optimization baseline (BO reuse must not
   change the math — same kernels, same inputs, only the upload is skipped).
   Log `max_abs / max_rel` informational (match the BF16 convention from
   `phase-1-kernel-validation`; do not use a tight `rtol`).
2. `make verify` still PASSES (the end-to-end gate; token-set top-k vs HF bf16).
3. Measured host/wall time is strictly lower than the baseline.

If (1)/(2) regress (NaN or garbage on the 2nd+ call, correct on the 1st) →
the BO bookkeeping is wrong; invoke `debug-bo-corruption`.
If (3) shows no gain → the per-call upload wasn't the bottleneck here;
document and keep or revert.

## Knowledge base references

- `programming_examples/llms/llama_kernel_builder/cache.py` —
  `KernelCache.load_and_run`, the `static_input_indices` /
  `intermediate_indices` mechanics this skill drives.
- `programming_examples/llms/llama32_1b/multi_launch_builder/*` — the worked
  example of weight + intermediate BO slots passed to a fused ELF.
- `programming_examples/llms/llama32_1b/llama32_1b_inference.py` —
  `prepare_runtime` / setup where per-layer BOs are allocated once.

## Workflow

### Step 1: Identify the BO slots

From the kernel group's argument signature, classify each BO slot:

- **weight / LUT slots** — written once, read every call → B1 candidates
  (`static_input_indices`).
- **kernel-overwritten slots** (the kernel's outputs and scratch) → B2
  candidates (`intermediate_indices`).
- **genuine per-call inputs** (the activation that changes each call) — leave
  as normal host-written inputs.

### Step 2: Pre-load weight BOs once (B1)

- Allocate per-layer weight BOs in `prepare_runtime()` (setup), keyed per
  layer (e.g. `bo_key=f"kernel_L{layer_idx}"`), and write the weights ONCE.
- On every `cache.load_and_run()`, pass `static_input_indices=[<weight slots>]`
  so the runtime does not re-upload them.

### Step 3: Reuse intermediate BOs (B2)

- For each slot the kernel fully overwrites, pass
  `intermediate_indices=[<output slots>]` on `cache.load_and_run()` so the
  host does not write that buffer before the call.

### Step 4: prefill vs decode

Same mechanic, two contexts (pass which one as the caller's parameter):

- **prefill**: weights re-used across the 16 per-layer calls within one pass.
- **decode**: weights re-used across every generated token × every layer —
  the win is much larger (16 layers × ~7 weight tensors × N tokens of upload
  removed). Static weight BOs are the dominant decode host-side optimization.

### Step 5: Validate + measure

- Run with BO reuse; compare output to the pre-reuse baseline → cosine ≥ 0.99.
- Re-run `make verify` → must still PASS.
- Profile host/wall time → must be strictly lower.

## Failure modes

| Symptom | Likely cause | Where to look |
|---|---|---|
| Correct on 1st call, NaN/garbage on 2nd+ call | per-layer BO key collision OR `static_input_indices` slot list wrong | Invoke `debug-bo-corruption` |
| Output mismatch on the very 1st call | a slot marked `intermediate` is actually read before being written | Re-classify that slot as a real input (drop it from `intermediate_indices`) |
| No host-time reduction | the per-call upload wasn't the bottleneck (kernel-bound) | Document; the merge skill (dispatch) or opt-layout-alignment may be the bigger win |

For any failure not in the table, invoke `superpowers:systematic-debugging`.

## Update protocol

Append to `<model>/docs/development_progress/phase{4,5}_*.md`:

```
## Buffer-object reuse
- B1 weight BOs: applied / skipped (reason)
- B2 intermediate BOs: applied / skipped (reason)
- Host/wall time before: X ms
- Host/wall time after:  Y ms
- Cosine vs baseline: <value>  | make verify: PASS/FAIL
```