Gpu Utilization Monitoring

GPU utilization monitoring

The number everyone quotes, nvidia-smi GPU utilization, only means "a kernel was resident during the last sample." A GPU pinned at 100 percent util can be running one tiny kernel and idling most of its SMs. To find real waste you need occupancy and throughput metrics, not the headline gauge, and DCGM exposes them.

Method

1. Stop trusting GPU-Util alone. DCGM_FI_DEV_GPU_UTIL is time-based: the fraction of samples with any active kernel. It says nothing about how many SMs ran. A memory-bound or launch-latency-bound loop reads 100 percent while doing a fraction of the FLOPs. Treat it as "not idle," never as "working hard."
2. Measure SM activity and occupancy. Pull DCGM_FI_PROF_SM_ACTIVE, the fraction of SMs with a resident warp, and DCGM_FI_PROF_SM_OCCUPANCY, resident warps against the maximum. SM_ACTIVE well below GPU_UTIL means the card is busy running work that occupies few SMs: kernels too small or the grid undersized.
3. Watch the tensor cores and memory pipes. For mixed-precision training the metric that matters is DCGM_FI_PROF_PIPE_TENSOR_ACTIVE; on an A100 or H100 doing real matmuls it should sit around 0.4 to 0.8. Compare it with DCGM_FI_PROF_DRAM_ACTIVE: high DRAM and low tensor means memory-bound, and no util figure changes that.
4. Deploy dcgm-exporter into Prometheus. Run dcgm-exporter as a DaemonSet, scrape into Prometheus, and dashboard in Grafana per GPU, per node, and per job. Tag each metric with the job or pod so idle GPUs have an owner. Alert when a GPU is allocated but SM_ACTIVE stays near zero for more than 10 minutes: that is paid-for silicon doing nothing.
5. Hunt the specific waste patterns. Allocated-but-idle, claimed with SM_ACTIVE near zero, means reclaim or right-size. High util with low tensor-active is a kernel or batch-size problem, not a capacity one. Power draw far under TDP (DCGM_FI_DEV_POWER_USAGE) confirms a coasting card, and step-time spread across ranks points at a straggler.
6. Correlate metrics with application throughput. Tie the GPU counters to tokens per second or samples per second and to watts. The real target is throughput per GPU and per watt, not any single gauge. A change that lifts tensor-active and throughput together is a win; one that only moves GPU-Util is noise.

Signals

• For a training job, is PIPE_TENSOR_ACTIVE high, or is GPU-Util masking near-idle tensor cores?
• Can you list GPUs allocated to a job but sitting at SM_ACTIVE near zero right now?
• Does power draw track the workload, or are cards coasting far below TDP while marked busy?
• Is every GPU metric tagged to an owner, so each pool of waste has a name?

Boundaries

This tells you a GPU is underused and roughly why; it does not fix the kernel (see cuda-kernel-basics) or the input pipeline. Fleet right-sizing and break-even math belong to cost planning (gpu-cost-planning), and reclaiming idle allocations is enforced by the scheduler (gpu-cluster-scheduling).