Gpu Inference Benchmark Eval

gpu-inference-benchmark-eval

DEEP-GAP: Deep-learning Evaluation of Execution Parallelism in GPU Architectural Performance — Palaniappan (2026) (arXiv:2604.14552, 2026)

What this evaluates

Evaluates GPU inference performance across different neural network models, numerical precision modes, and batch sizes. It measures how architectural differences and execution parallelism impact throughput, latency, and memory utilization under production-like conditions.

Datasets

• ResNet models (ResNet-18, ResNet-50, ResNet-101) with synthetic inputs — total ?; splits: test (-1); repo https://github.com/kpalania1/deep-gap-gpu-inference-benchmark

Metrics

• throughput (images/sec) (primary) — range: other
  • Total number of images processed divided by the total inference time over timed iterations. Calculated as (batch_size * timed_iterations) / sum(latencies).
• median latency — range: other
  • The middle value of the distribution of per-inference execution times recorded over timed iterations.
• P99 latency — range: other
  • The 99th percentile of per-inference execution times, capturing tail latency behavior.

Input / output format

Input: GPU input tensors matching the model's expected input shape (e.g., [batch_size, 3, 224, 224] for ResNets), allocated on the target device with gradient computation disabled.

Output: Inference output tensors and performance logs containing per-iteration latency, computed median/P99 latency, throughput (images/sec), and peak GPU memory usage.

Scoring recipe

latencies = []
for _ in range(repeats):
    for _ in range(warmup): run_inference()
    for _ in range(timed):
        t0 = time.perf_counter()
        run_inference()
        latencies.append(time.perf_counter() - t0)
median_lat = np.median(latencies)
p99_lat = np.percentile(latencies, 99)
throughput = (timed * batch_size) / sum(latencies)

Common pitfalls

• Skipping the warm-up phase, which causes cold-start overhead (kernel init, cache population) to skew latency measurements.
• Not fixing PyTorch inter-op threads to 1, allowing CPU scheduling variability to contaminate GPU execution metrics.
• Evaluating only a single batch size, missing critical saturation points and compute/memory-bound regime shifts.

Evidence (verbatim from paper)

For each configuration, we record median latency, mean latency, standard deviation, and P99 latency to capture both average and tail behavior. In addition, we measure throughput in images per second and track peak GPU memory usage using NVML.

Citation

@misc{palaniappan2026deepgap,
  title={DEEP-GAP: Deep-learning Evaluation of Execution Parallelism in GPU Architectural Performance},
  author={Palaniappan (2026)},
  year={2026},
  note={arXiv:2604.14552}
}

• arXiv: 2604.14552