cuTile optimization skill, wiki assisted
For human users: how to request kernel optimization
This skill is a reference the agent consults when it has a concrete question —
not an optimization procedure to invoke by name. Do not prompt
"use the tilegym-cutile-optimizing skill to optimize this kernel"; forcing
the skill to load up front defeats its gate and measurably hurts results on
kernels the wiki covers thinly. Instead, state only the goal — preferably via
/goal, for example:
/goal Optimize the cuTile implementation of <op> in this repo.
Correctness must keep passing: pytest tests/ops/test_<op>.py --backend cutile
Performance is measured by: python tests/benchmark/bench_<op>.py
Work in a scratch space and leave the original implementation untouched.
Deliver your best correct version as a ready-to-apply patch or file copy in
the scratch space, with measured before/after results, for human review.
The agent loads this skill mid-session on its own when a plan-changing question forms.
The wiki is the repository's canonical cuTile knowledge wiki at
wiki/cutile-knowledge-wiki/; all wiki paths below are relative to the
repository root.
The wiki contains prior experience from other kernels, machines, toolchains, and sessions. Use it to answer a specific question. Do not browse it just because it is available.
Core rule
Think first. Query second. Measure everything.
Before reading an optimization page:
- Inspect the current kernel, baseline results, profile data, and your own experiment notes.
- Form your own diagnosis and next-step plan.
- State one concrete question whose answer could change that plan.
- Check whether your earlier experiments or measurements already answer it.
- Open the wiki only if the question remains unresolved.
Use this short checkpoint:
Current diagnosis:
Planned next move:
Unresolved question:
What answer would change the plan:
If you cannot fill in Unresolved question, continue independent analysis instead
of reading the wiki.
Language and API correctness questions are exempt from this gate: check exact semantics whenever uncertainty would otherwise produce invalid code.
Route 0 — language or API question
Use this route for questions about:
- cuTile execution and launch behavior
- tile-space indexing
ct.loadversusct.gather- padding and masking
- numerical behavior
- traced Python restrictions
- compiler hints
- wrapper, specialization, and caching behavior
- exact API availability
Use the official cuTile Python documentation for the formal API reference: https://docs.nvidia.com/cuda/cutile-python. The source and its examples are at https://github.com/NVIDIA/cutile-python.
Use the wiki language page for the practical working model, common conversion mistakes, and known failure behavior:
grep -ril "id: cutile-language" wiki/cutile-knowledge-wiki/
Ask a narrow question. For example:
Does this runtime page index require gather for the whole tensor, or can I gather
the scalar page id and then use ct.load?
Do not use optimization technique pages to answer basic language legality questions.
Route 1 — kernel shape to kernel-family page
Use this route only after forming your own opening design or obtaining a working candidate.
Use it to answer questions such as:
- What usually dominates performance for this kernel family?
- Which implementation choices may be missing from my current design?
- Is there a known alternative architecture worth testing?
Read only the relevant parts of:
What dominates performanceApplicable techniques
Do not copy the page as the opening design. Compare it against the design you formed independently.
| Looks like | Page |
|---|---|
| dense matmul / bmm / grouped GEMM | wiki/cutile-knowledge-wiki/kernels/kernel-gemm.md |
| attention prefill (FMHA forward) | wiki/cutile-knowledge-wiki/kernels/kernel-attention-prefill.md |
| attention decode (few queries, long KV) | wiki/cutile-knowledge-wiki/kernels/kernel-attention-decode.md |
| layer norm / RMS norm / per-slice stats | wiki/cutile-knowledge-wiki/kernels/kernel-norms.md |
| row softmax, forward or backward | wiki/cutile-knowledge-wiki/kernels/kernel-softmax.md |
| RoPE / paired 2D rotations | wiki/cutile-knowledge-wiki/kernels/kernel-rope.md |
| prefix sum / cumsum / histogram | wiki/cutile-knowledge-wiki/kernels/kernel-scan-histogram.md |
| embedding / index-select / paged gather | wiki/cutile-knowledge-wiki/kernels/kernel-gather-scatter.md |
| MoE token-routing auxiliaries | wiki/cutile-knowledge-wiki/kernels/kernel-moe-align.md |
A family match is only a starting point. A recommendation may also depend on the current architecture, operation role, shape, dtype, GPU, and toolchain.
Route 2 — measured symptom to pattern page
Use this route only when measurements support a specific symptom.
Available pattern pages include:
wiki/cutile-knowledge-wiki/patterns/pattern-memory-bound.mdwiki/cutile-knowledge-wiki/patterns/pattern-compute-bound.mdwiki/cutile-knowledge-wiki/patterns/pattern-low-sm-utilization.mdwiki/cutile-knowledge-wiki/patterns/pattern-register-pressure.mdwiki/cutile-knowledge-wiki/patterns/pattern-tail-effect.mdwiki/cutile-knowledge-wiki/patterns/pattern-host-overhead-bound.mdwiki/cutile-knowledge-wiki/patterns/pattern-shape-heterogeneity.md
Before opening a pattern page, state the evidence:
Observed symptom:
Measurement supporting it:
Current suspected cause:
Question for the wiki:
Treat the page's candidate techniques as a menu of hypotheses, not a checklist.
Do not try them in page order by default. First check whether the stated cause actually matches the current code and profile.
For example:
- Do not choose TMA only because the kernel is memory-bound.
- First determine whether the tile is reused, streamed once, block-aligned, or genuinely random.
- Do not tune a memory-bound kernel that is already close to peak bandwidth without first considering whether fewer bytes must be moved.
A plateau by itself is not a query. Identify a concrete unknown before reading.
Route 3 — check a specific technique before applying it
Use this route when you already plan to test a specific technique and need to check its conditions, parameters, or known failure cases.
Technique pages include:
wiki/cutile-knowledge-wiki/techniques/tech-tile-size.mdwiki/cutile-knowledge-wiki/techniques/tech-tma-load.mdwiki/cutile-knowledge-wiki/techniques/tech-num-ctas.mdwiki/cutile-knowledge-wiki/techniques/tech-latency-hint.mdwiki/cutile-knowledge-wiki/techniques/tech-occupancy.mdwiki/cutile-knowledge-wiki/techniques/tech-persistent-grid.mdwiki/cutile-knowledge-wiki/techniques/tech-epilogue-fusion.mdwiki/cutile-knowledge-wiki/techniques/tech-group-swizzle.mdwiki/cutile-knowledge-wiki/techniques/tech-ftz-approx.md
Read only the sections needed to answer the question:
What it isWhen to useCaveats- the evidence closest to the current case
Before applying the advice, compare:
- the role of the operation, such as mainloop load, epilogue load, or store
- the surrounding kernel architecture
- shape and dtype
- register pressure
- GPU architecture
- cuTile and TileIRAS versions
- the code state in which the historical result was measured
Treat caveats as scoped evidence, not universal bans.
A result measured in one architecture does not prove that the same technique will have the same effect in another architecture.
Fallback search
Use fallback search only for an exact term, error message, API, kernel type, or technique:
grep -ril "<specific term>" wiki/cutile-knowledge-wiki/
The full page inventory is:
wiki/cutile-knowledge-wiki/index.md
Do not use directory listings or broad searches as onboarding.
Do not search for generic phrases such as:
how to optimize this kernel
best techniques
what should I try
Read one substantive page per question by default. Open a second page only when the first page creates a concrete follow-up question.
After reading
Immediately record:
New information:
Why it applies, or may not apply:
Plan before reading:
Plan after reading:
Minimal experiment:
If Plan after reading is unchanged, stop. Do not keep browsing and do not perform
an experiment merely to comply with the skill.
If the wiki confirms something already present in the plan, treat the read as confirmation, not as a new optimization direction.
If the page conflicts with a current-board measurement, trust the current measurement while preserving the exact context of both results.
API and toolchain check
Before investing in a recipe that uses a specific API or compiler feature:
- Verify that the current cuTile and TileIRAS versions support it.
- Check whether the page provides a compatible fallback.
- If version support is unclear, run the smallest possible compile check first.
- If the feature requires a newer version, mark the recipe inapplicable to this environment and continue with another approach.
A recipe requiring a newer toolchain is not evidence that the current platform is broken.
Do not let an unavailable preferred path prevent testing a compatible fallback.
Running the experiment
When wiki information changes the plan:
- Make one controlled change where possible.
- Compare against the direct parent candidate.
- Run correctness checks.
- Benchmark on the current board.
- Keep, revise, or revert based on the result.
Do not apply several wiki suggestions at once unless they form one inseparable design. Otherwise, the result cannot show which suggestion helped.
Work in a scratchpad, not in place: keep experimental kernel variants, one-off
harnesses, and notes in a scratch directory (for example sandbox/ in the
working tree, or /tmp). The original implementation stays untouched end to
end — verify a candidate by temporarily applying it and restoring the original
afterward. Deliver the final best-effort kernel as a ready-to-apply patch or
file copy in the scratch directory, together with measured before/after
results on the official benchmark; whether it replaces the original is a human
decision, not the agent's.
Workflows
New kernel
- Inspect the task and form an independent v1 design.
- Use Route 0 immediately if language or API semantics are unclear.
- Before reading optimization pages, write down the independent design.
- Query Route 1 only with a concrete question that may change that design.
- Record any plan change caused by the page.
- Implement and measure.
Working kernel
- Inspect the current code and measurements.
- Identify one likely bottleneck or uncertain decision.
- Check your earlier measurements and experiment notes.
- Use Route 1, 2, or 3 for the exact unresolved question.
- Run one minimal A/B experiment.
- Record the result with its measured scope (shape, dtype, GPU, code state).
Plateaued kernel
- Do not browse the wiki merely because progress stopped.
- Review failed experiments and current measurements.
- Identify a specific missing fact, disputed assumption, or untested alternative.
- Query only that question.
- If no specific question exists, continue independent exploration.
Specific optimization already planned
- State the intended change and expected reason.
- Use Route 3 to check conditions and caveats.
- Verify toolchain support.
- Record whether the page changed the plan.
- Run the smallest useful experiment.
Doctrine
Every number, configuration, and recommendation in the wiki is prior evidence from another kernel, board, toolchain, code state, or day.
- Re-measure on the current system.
- Trust current measurements when they disagree with historical evidence.
- Do not treat a page as an instruction to use a technique.
- Do not treat a caveat as a ban outside its measured scope.
- Do not let the wiki choose your entire search space.
- Profiler-derived, first-principles, and novel approaches are always allowed.
- The absence of a technique from the wiki is not evidence against trying it.
- The wiki is useful only when it supplies information that changes a concrete decision.