Profile Model

Profile a model on MAX

This skill answers three questions about any model you can run on MAX, whether it's a built-in architecture or a custom one loaded with --custom-architectures:

1. Is my GPU actually being used? (utilization, memory, clocks)
2. Where does my model spend the most time? (which kernels / phases)
3. Why is this kernel slow? (occupancy, stalls, roofline)

It works from a pip or pixi install of MAX, driving the public max CLI or a Python script.

How to decide what to run

Each question needs a deeper, more expensive capture than the last. Work from the cheapest check toward the most invasive, and let each result decide whether going deeper is even worth it — don't capture more than the question needs.

1. Start with the utilization check. If the GPU is idle or lightly loaded while the workload runs, the bottleneck is host-side (CPU, transfers, small batch) — stop and report that. A kernel trace won't tell you anything a busy GPU wouldn't.
2. If the GPU is busy, capture a kernel breakdown. This is the common case and usually the final answer: it shows which kernels dominate GPU time.
3. Only if one kernel dominates and you need to know why, do a single-kernel deep dive on that one kernel. Skip this unless a breakdown has already pointed at a specific kernel — it replays the kernel many times and is slow.

Read the reference for the step you're on rather than loading all of them:

• Utilization check → references/utilization-api.md (and the bundled scripts/gpu_snapshot.py)
• Kernel breakdown, NVIDIA → references/nsys.md
• Kernel breakdown, AMD → references/rocprofv3.md
• Single-kernel deep dive (NVIDIA) → references/ncu.md

Models loaded with --custom-architectures

Profiling itself works the same for a custom architecture as for a built-in one. What differs is loading: a custom architecture has extra prerequisites and failure modes — import compatibility with your installed MAX version, PYTHONPATH traps, architecture-name collisions, base models with no chat template, and a max serve limitation for custom embeddings models. Those stop the model from loading before any profiling can run. If you're profiling a model passed via --custom-architectures, read references/custom-architectures.md first.

Reading the result (what the kernel mix tells you)

Profiling answers two things: is the GPU saturated (the utilization check) and where does time go (the kernel breakdown). Map what you see back to a diagnosis:

• Decode dominated by gemv_split_k_* GEMVs, with *_mha_* attention and rms_norm below — this is the normal, healthy shape for a dense LLM token generation (decode is memory-bound GEMV). Nothing to chase.
• Sparse-MoE decode dominated by grouped / block-scaled matmul plus routing kernels — the per-expert grouped matmul (*grouped* / block_scaled_* matmul) is the dominant kernel instead of a dense GEMM, alongside top-k routing (topk_*, moe_create_indices) and expert gather/scatter; on multi-GPU, EP dispatch/combine collectives appear too. This is the healthy shape for a Mixture-of-Experts model — routing and gather overhead is expected, and only worth chasing if it rivals the matmul itself.
• Low-bit weights (FP8 / FP4 / NVFP4) add quant-prep kernels — alongside the matmul you'll see dynamic activation-quantization and scale-layout kernels (*quantize*, *block_scales*) and block-scaled matmul variants rather than plain gemv/gemm. Time spent in quant/dequant prep is normal for a low-bit model; flag it only if it dwarfs the matmul it feeds. Kernel prefixes are GPU-arch-specific (sm100_* on Blackwell, different on Hopper / MI) — match on the kernel family, not the exact name.
• A custom architecture not hitting the expected fused kernel (for example attention showing as generic elementwise/matmul instead of a *_mha_* kernel, or norms/RoPE unfused) — a wiring signal: the graph may not be built the way you think, even if logits pass. Worth a look during bring-up.
• Pathologically low decode tok/s + an outsized prefill-shaped kernel mix repeating every step — classic O(n²) re-prefill (no/broken KV cache). This is a correctness-adjacent bug, not just slowness.
• Low gpu_usage_percent while a workload runs — host-bound, launch/sync overhead, or batch too small; the GPU isn't the bottleneck.
• Throttle reasons present — the GPU is clock-limited; numbers aren't representative until you address thermal/power.

The durable algorithm

Tooling and MAX packaging change often; this shape is the part worth preserving. Whichever capture you run, follow it:

1. Detect the environment before committing to a tool. Run nvidia-smi or rocm-smi to learn the vendor — the kernel-breakdown and deep-dive tools differ by vendor. Confirm the profiler is installed (which nsys / which rocprofv3 / which ncu); if it's missing, tell the user the exact install line (see the reference) rather than failing midway.
2. Confirm before any long-lived run. Profiling max serve warms a compile cache (cold compile can take minutes), launches a server, and runs a benchmark. Before doing that, show the user the model, the flags, and the planned commands, and wait for confirmation. A one-shot max generate --profile on a tiny model is cheap enough to skip this.
3. Warm caches first, profile second. The first run of a model pays one-time compile and weight-load costs that drown out the real kernel time. Do an unprofiled warm-up run, then the profiled run, so the capture reflects steady-state inference — not compilation.
4. Keep the capture window small. A 10-second window at concurrency 1 is enough to see the kernel mix. Long captures produce huge trace files that are slow to open and no more informative.
5. Fail fast. After launching a server, check within a few seconds that the process is alive and the log has no early error (model not found, OOM, bad flag) before you start polling for /health. Don't wait out a full timeout on a server that already died.
6. Verify the artifact, then report. Confirm the output file exists and is non-trivial in size before claiming success. Report the artifact path, how to open it, and the headline numbers (top kernels, or peak GPU utilization).
7. Clean up only what you started. Kill the server / benchmark process group you launched. Avoid broad pkill -f max on a shared box — you may stop someone else's run.

Nightly vs stable

Profiling features land in nightly before stable, so this skill targets the nightly max-serve build. Install it with pixi:

pixi init quickstart -c https://conda.modular.com/max-nightly/ -c conda-forge
cd quickstart && pixi add max-serve
pixi run max --version          # expect a *.dev build

The utilization API (GPUDiagContext, BackgroundRecorder) ships in the conda max package, so install via pixi or conda rather than a plain pip install max wheel, which may not expose max.profiler.gpu. If a command below 404s or an import fails, confirm you're on a recent nightly build.

Install notes

• MAX itself: a project with the nightly max-serve package installed via pixi (see Nightly vs stable above). All max CLI commands below assume you can run pixi run max ... in that project. The max-serve package includes extra dependencies required to serve a model with MAX, which aren't included with the main max package.
• nsys and ncu ship with the CUDA Toolkit. If which nsys fails but CUDA is present, export PATH=/usr/local/cuda/bin:$PATH. Otherwise install Nsight Systems / Nsight Compute from NVIDIA, or sudo apt-get install nsight-systems.
• rocprofv3 ships with ROCm (/opt/rocm/bin). Add it to PATH if needed.
• The utilization API needs no extra tooling beyond max — it talks to NVML / ROCm SMI directly through max.profiler.gpu.

Fast paths

The lowest-friction commands, each expanded in its reference file:

• "Is my GPU being used?" → run scripts/gpu_snapshot.py alongside a short benchmark. Pure Python, works on NVIDIA and AMD.
• "Give me a kernel breakdown, fast." → pixi run max generate --model <m> --prompt "hello" --num-warmups 1 --profile. Prints a ranked top-N GPU-kernel table and writes a .nsys-rep (NVIDIA). Falls back to a CPU summary without a GPU.
• "Profile my serving benchmark." → the serving-endpoint flow with MODULAR_ENABLE_PROFILING=detailed + nsys launch / rocprofv3.