Gnn Op Runtime Benchmark

gnn-op-runtime-benchmark

Operation-Level Performance Benchmarking of Graph Neural Networks for Scientific Applications — Hosseini et al. (2022) (arXiv:2207.09955, 2022)

What this evaluates

Evaluates the runtime and memory efficiency of low-level Graph Neural Network and sparse tensor operations on NVIDIA A100 GPUs. It probes how input sparsity, tensor dimensions, and reduce factors affect computational overhead when operations are pushed to near-full GPU memory capacity.

Datasets

• Synthetic Tensors (varying sparsity & dimensions) — total ?; splits: (unstated)

Metrics

• median_runtime (primary) — range: seconds
  • Median execution time in seconds over at least 14 benchmark iterations. Only the forward pass is measured. Validity requires an interquantile range ≤ 0.00017 seconds across single iterations.

Input / output format

Input: Torch tensors (float32 for data, int64 for indices) with varying dimensions, sparsity levels, and reduce factors. Input sizes are scaled to utilize ≥95% of the 40GB GPU memory.

Output: Runtime measurements in seconds (median of ≥14 runs) and memory utilization percentages for each operation under tested configurations.

Scoring recipe

def compute_metric(runs):
    assert len(runs) >= 14
    sorted_runs = sorted(runs)
    median_time = sorted_runs[len(sorted_runs) // 2]
    iqr = max(runs) - min(runs)
    if iqr > 0.00017:
        raise ValueError('Benchmark unstable: IQR exceeds threshold')
    return median_time

Common pitfalls

• Only the forward pass is benchmarked; backward pass overhead is completely ignored.
• Input sizes are fixed to hit ≥95% GPU memory capacity, which may not reflect typical real-world batch sizes or sparsity levels.
• Sparsity variations are tested at a fixed tensor size, meaning sparser inputs will not utilize full memory capacity, potentially skewing memory-bound vs compute-bound conclusions.

Evidence (verbatim from paper)

We test each operation at the full memory capacity of the machine in order to facilitate fair comparison of operations across different hardware platforms. All operations where benchmarked at least 14 times and all reported value are the median value benchmarked. All operations benchmarked have an interquantile range of ≤ 0.00017 seconds across single benchmark iterations.

Citation

@misc{hosseini2022gnnoperationbenchmark,
  title={Operation-Level Performance Benchmarking of Graph Neural Networks for Scientific Applications},
  author={Hosseini et al. (2022)},
  year={2022},
  note={arXiv:2207.09955}
}

• arXiv: 2207.09955