Sparsetir Eval

sparsetir-eval

SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning — Ye et al. (2022) (arXiv:2207.04606, 2022)

What this evaluates

Evaluates the runtime performance and speedup of sparse deep learning operators (SpMM, SDDMM) and end-to-end models (GraphSAGE, RGCN, Transformers) on GPU hardware using composable sparse formats and transformations. It probes how format decomposition and modular scheduling primitives improve cache utilization, load balancing, and Tensor Core utilization compared to vendor libraries and existing compilers.

Datasets

• cora — total ?; splits: test (-1)
• citeseer — total ?; splits: test (-1)
• pubmed — total ?; splits: test (-1)
• ppi — total ?; splits: test (-1)
• ogbn-arxiv — total ?; splits: test (-1)
• ogbn-proteins — total ?; splits: test (-1)
• reddit — total ?; splits: test (-1)

Metrics

• speedup (primary) — range: ratio
  • Ratio of baseline execution time to SparseTIR execution time (T_baseline / T_sparsetir). Geometric mean is used for aggregation across feature sizes or datasets.

Input / output format

Input: Sparse adjacency matrices (in CSR, BSR, or hyb formats) and dense feature matrices representing graph or attention workloads.

Output: Execution time per operator or training epoch, reported as a normalized speedup ratio relative to baseline implementations.

Scoring recipe

def compute_speedup(baseline_time, sparsetir_time):
    return baseline_time / sparsetir_time

def geometric_mean_speedup(speedups):
    import math
    return math.exp(sum(math.log(s) for s in speedups) / len(speedups))

# Example usage:
# speedups = [compute_speedup(t_base, t_sparse) for t_base, t_sparse in timings]
# final_metric = geometric_mean_speedup(speedups)

Common pitfalls

• Padding zeros in composable formats increases FLOPs but improves runtime due to better scheduling and compile-time load balancing.
• Numerical accuracy must be verified against existing frameworks before trusting speedup gains.
• Results are hardware-specific (V100 vs RTX 3070) and not portable across architectures.

Evidence (verbatim from paper)

We measure the geometric mean speedup of different SpMM implementations against cuSPARSE for feature size d∈{32,64,128,256,512}.

Citation

@misc{ye2022sparsetir,
  title={SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning},
  author={Ye et al. (2022)},
  year={2022},
  note={arXiv:2207.04606}
}

• arXiv: 2207.04606