Migperf Eval

migperf-eval

MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs — Huaizheng Zhang et al. (2023) (arXiv:2301.00407, 2023)

What this evaluates

Evaluates deep learning training and inference workloads on NVIDIA's Multi-Instance GPU (MIG) technology. It systematically measures performance trade-offs across different partition sizes, batch sizes, and model architectures, while comparing MIG against software-based GPU sharing (MPS) and testing framework compatibility.

Datasets

• Representative DL Models (ResNet18, ResNet50, BERT, ViT, Diffusion) — total ?; splits: test (-1); repo https://github.com/MLSysOps/MIGProfiler

Metrics

• average latency — range: ms
  • Average processing time per batch of requests for inference workloads.
• tail latency (primary) — range: ms
  • X-th percentile (e.g., 99th) of request latencies, measuring worst-case processing time.
• throughput — range: req/min
  • Number of training samples or inference requests processed per unit time (e.g., per minute).
• GRACT — range: [0, 1]
  • Ratio of allocated computation resources to total available resources on a GPU instance.
• Frame Buffer — range: GB
  • Occupied GPU memory during workload execution.
• Energy Consumption — range: other
  • Estimated electricity used for running a workload over a specific period (e.g., 5 minutes).

Input / output format

Input: Deep learning training or inference workloads specified by model architecture, batch size, and MIG partition configuration (e.g., 1g.10gb).

Output: Performance metrics (latency, throughput, utilization, memory, energy) and framework compatibility status (detected/not detected).

Scoring recipe

for each workload_config in configs:
    start_timer()
    run_inference_or_training(workload_config)
    end_timer()
    metrics[workload_config] = {
        'avg_latency': mean(request_latencies),
        'tail_latency': percentile(request_latencies, 99),
        'throughput': count_requests / duration_minutes,
        'gract': compute_gpu_utilization(),
        'fb': measure_gpu_memory(),
        'energy': estimate_energy_consumption(duration_minutes)
    }
return metrics

Common pitfalls

• Performance comparison between MIG and MPS is highly sensitive to batch size; small batches show negligible differences, while large batches reveal MIG's tail latency advantage.
• Framework compatibility tests only check if frameworks can detect multiple GIs; they do not evaluate actual performance on non-zero GIs due to driver limitations.
• Energy consumption is driven by processing speed rather than just batch size; larger instances may consume less energy for the same workload due to faster completion.

Evidence (verbatim from paper)

Latency measures the processing speed. We use two metrics, average latency and tail latency for inference workloads. Average latency measures the average processing time for a batch of requests, while tail latency means that the X-th (e.g. 99) percentile of requests has lower latency than the given value. Throughput measures the training samples or inference requests that a device can process within a time unit (e.g., 1 min).

Citation

@misc{zhang2023migperf,
  title={MIGPerf: A Comprehensive Benchmark for Deep Learning Training and Inference Workloads on Multi-Instance GPUs},
  author={Huaizheng Zhang et al. (2023)},
  year={2023},
  note={arXiv:2301.00407}
}

• arXiv: 2301.00407