1---2name: gpu-architecture-fundamentals3description: This skill should be used when reasoning about GPU architecture fundamentals to guide kernel optimization choices such as memory hierarchy usage, execution model mapping, block sizing, and latency-aware tuning across HIP, Triton, and PyTorch.4metadata:5 source: https://skillmd.com/skills/wenyi-li/gpu-architecture-fundamentals6---78# GPU Architecture Fundamentals910## Purpose11- Reference core GPU concepts (memory hierarchy, execution model) and typical bandwidth/latency numbers to ground optimization choices.12- Provide block size heuristics and ready-to-use checklists before writing or tuning kernels.13- Map common optimization patterns across HIP, Triton, and PyTorch to pick framework-specific tactics quickly.1415## When to Use16- Planning or reviewing kernel designs where occupancy, memory bandwidth, or latency hiding are concerns.17- Selecting grid/block shapes, deciding on shared memory usage, or checking for coalesced accesses.18- Comparing optimization levers across frameworks when porting kernels.1920## How to Use21- Recall memory hierarchy: prefer registers > shared/L1 > L2 > HBM; treat HBM as ~400–800 cycle latency, registers ~0, shared ~20–30 cycles.22- Anchor bandwidth sense-checks with table values (e.g., MI300X HBM3 ~5.3 TB/s, A100 HBM2e ~2.0 TB/s).23- Choose block sizes by operation: element-wise 256–1024 threads, reduction 256–512, matmul tiles 128x128 or 256x128, conv 32x32 or 64x64.24- Apply execution model mapping: thread ↔ element/partial tile, warp/wavefront ↔ contiguous data segments, block/workgroup ↔ tiles sharing shared memory, grid ↔ full problem coverage.25- Run the optimization checklist before finalizing kernels:26 - Ensure coalesced and vectorized memory access; avoid shared memory bank conflicts.27 - Target occupancy >50%; watch register pressure and shared memory usage to avoid spilling.28 - Fuse operations where possible; leverage mixed precision when valid.29 - Overlap transfers with compute; tune block/grid dimensions; unroll small loops.30- Use pattern summaries to pick tactics per framework:31 - Memory: HIP manual strides/shared, Triton `tl.arange`/implicit tiling, PyTorch `.contiguous()`/compiler.32 - Compute: HIP manual fusion/unroll, Triton `@triton.jit` + `tl.constexpr`, PyTorch `torch.compile`/FlashAttention.33 - Parallelism: HIP block/grid + occupancy APIs, Triton autotune + constexpr block sizes, PyTorch compiler/automatic launch config.3435## Quick Checks36- If performance regresses, compare achieved block size and occupancy to table heuristics.37- If L2/HBM traffic is high, add tiling or fusion; if shared memory stalls, check bank conflicts and tile padding.38- When switching hardware, re-evaluate bandwidth and latency assumptions and retune block sizes accordingly.