B200 Kernel Roofline Triage

B200 Kernel Roofline Triage

R — Source evidence (Reading, paraphrased)

• [S3] Split the kernel's ceiling into a "compute roof" and a "bandwidth roof", and use arithmetic intensity to decide which side is more likely the current constraint.
• [S3] For low-arithmetic-intensity operators, prioritize reducing bytes, fusion, reuse, or a narrower dtype; for high-arithmetic-intensity GEMMs, the focus is keeping the Tensor Cores continuously busy.
• [S16] Blackwell still inherits the general CUDA best practices: parallelize, reduce Host↔Device transfers, coalesce accesses, and reduce redundant accesses and warp divergence.

Source: distilled from "Modern GPU Programming for MLSys" (https://mlc.ai/modern-gpu-programming-for-mlsys/) and the NVIDIA Blackwell tuning/compatibility guides. Short paraphrases only; no long passages are reproduced.

I — Methodology skeleton (Interpretation)

Do not start optimizing from "this trick is new"; answer three questions first:

1. How much useful compute is done per output element? Estimate the FLOPs.
2. How many bytes were moved, from which level of storage, to do that compute? Give at least the HBM accounting; add L2/SMEM accountings when necessary.
3. Does the current implementation actually convert the theoretical roof into hardware busyness? Even with high algorithmic arithmetic intensity, a wrong layout, serialized load/compute/store, resource pressure, or the launch shape can still leave the Tensor Cores idle.

The final diagnosis must not just say "memory-bound/compute-bound"; it must also give the evidence, the alternative explanations not yet ruled out, and the next minimal experiment.

A1 — Applications in the source (Past Application)

Case 1: a large GEMM that should be compute-bound yet measures low

• Problem: the matrices are large enough that by arithmetic intensity it should sit to the right of the ridge under the compute roof, yet Tensor Core utilization is low.
• How the methodology was used: after ruling out HBM bandwidth, check whether execution is still serialized as "load→compute→store" and whether TMA, software pipelining, or warp specialization is missing.
• Conclusion: the bottleneck is not "move a little less HBM traffic" but idle compute engines and insufficient pipeline overlap.

Case 2: elementwise operators such as RMSNorm/GELU

• Problem: adding more math optimizations barely changes performance.
• How the methodology was used: their FLOPs/byte is very low, so first check coalescing, the number of reads/writes, fusion opportunities, and the dtype.
• Conclusion: the goal is to approach the bandwidth roof, not to chase Tensor Core peak.

A2 — Trigger scenarios (Future Trigger) ★

In what situations will the user need this skill?

1. "This kernel only hits xx TFLOPS on B200 — where do I look first?"
2. "Should this operator be fused, or switched to TMA/Tensor Cores?"
3. "Do a roofline diagnosis for me and give an ordered sequence of optimization experiments."

Language signals

• "This kernel only hits xx TFLOPS on B200 — where do I look first?"
• "Should this operator be fused, or switched to TMA/Tensor Cores?"
• "Do a roofline diagnosis for me and give an ordered sequence of optimization experiments."

Distinction from adjacent skills

Versus b200-gemm-optimization-ladder: this skill first determines the bottleneck and the optimization direction; the latter gives the step-by-step implementation route specifically for GEMM. Versus b200-layout-contract-auditor: this skill does global performance attribution; the latter audits addresses and hardware layout contracts in depth.

E — Executable steps (Execution)

Once the skill is activated, the agent must follow this procedure:

1. Collect the minimal fact set
   • The operator formula, shapes, dtypes, batch, and whether inputs/outputs are reused.
   • The current implementation path (CUDA/Triton/TIRx/CUTLASS/framework op), the timing methodology, warmup, and whether communication is included.
   • The B200 model/power limit/clocks, the compile target, and a profiler summary.
   • Done criterion: you can write a one-line estimate of "useful FLOPs" and a one-line estimate of "HBM bytes".
2. Compute at least one roofline accounting
   • AI_HBM = useful_FLOPs / HBM_bytes.
   • Prefer the measured bandwidth on the user's machine and the same-dtype peak; without them, only order-of-magnitude inference is possible — label the assumptions explicitly.
   • Done criterion: an initial memory-bound / compute-bound / near-ridge call.
3. Check whether the implementation contradicts the initial call
   • memory-bound: check repeated reads/writes, intermediate tensors spilled to HBM, uncoalesced accesses, an overly wide dtype, and insufficient request concurrency.
   • compute-bound: check Tensor Core instructions, tile utilization, TMA/compute/store overlap, warp roles, tail tiles, and small shapes.
   • Neither fits: check launch latency, synchronization, CPU submission, communication, power/frequency, and occupancy resource pressure.
4. Build the evidence matrix
   • For each candidate bottleneck write "supporting evidence / counter-evidence / measurement needed".
   • Done criterion: at least 3 candidates listed, and they must not all be the same class of micro-optimization.
5. Design the minimal falsifiable experiment
   • Change only one factor at a time, e.g. disable fusion, switch to a contiguous layout, increase the pipeline depth, use a fixed shape, lock the clocks.
   • Stopping rule: if the experiment result contradicts the hypothesis, go back to step 3; do not keep stacking the same class of optimization.
6. Output the optimization order
   • P0: correctness and timing credibility; P1: the roof-determined primary bottleneck; P2: secondary scheduling/resource issues; P3: fine-tuning.

Required outputs

1. Conclusion: the current choice/diagnosis, never a vague "we may need to look at everything".
2. Evidence or assumptions: which items come from user data and which are assumptions pending verification.
3. Contract/table/timeline: the auditable intermediate artifacts corresponding to this skill.
4. Minimal validation: a correctness test, a boundary test, and one falsifiable experiment.
5. Risks and fallback: the alternative path when hardware, version, or resource requirements are not met.

B — Boundaries (Boundary) ★

Do not use when

• The user only asks about B200 memory capacity, price, or rack specs, with no operator or performance question.
• With no shape, dtype, timing, or dataflow information at all, do not assert a bottleneck outright.

Failure modes

• Treating the theoretical peak as a directly achievable promise.
• Looking only at occupancy while ignoring that explicit pipelining can already hide the latency; or conversely, looking only at pipelining while ignoring that resources prevent residency in the first place.
• Substituting a single profiler percentage for end-to-end evidence.

Limitations

• Roofline has limited explanatory power for irregular accesses, short kernels, dependency chains, and cross-GPU communication; latency and communication models must be added when necessary.

Related skills

• depends-on: none
• contrasts-with: b200-gemm-optimization-ladder
• composes-with: b200-layout-contract-auditor, b200-tma-pipeline-designer, b200-warp-specialized-debugger

Audit info

• Validation passed: V1 ✓ / V2 ✓ / V3 ✓
• Test definitions: 6 (3 should_trigger / 2 should_not_trigger / 1 edge_case)
• Hardware validation: not performed; must be verified on a target B200
• Distilled: 2026-06-25