Tenspiler Eval

tenspiler-eval

Tenspiler: A Verified Lifting-Based Compiler for Tensor Operations (Extended Version) — Qiu et al. (2024) (arXiv:2404.18249, 2024)

What this evaluates

Evaluates the correctness and performance of a verified lifting-based compiler that transpiles sequential C++/Python code into tensor operations across various DSLs and hardware accelerators. It measures synthesis efficiency, kernel execution speedup, and end-to-end performance including data transfer overhead.

Datasets

• TENSPILER benchmark suite — total 69; splits: test (69)

Metrics

• kernel_performance (primary) — range: other
  • Ratio of baseline C++ execution time to translated code execution time, excluding data transfer overhead.
• end_to_end_performance — range: other
  • Ratio of baseline C++ execution time to translated code execution time, including initial setup and data movement overhead.
• synthesis_time — range: seconds
  • Wall-clock time taken by the synthesis and verification algorithm to generate a correct translation, capped at a 1-hour timeout.

Input / output format

Input: Sequential C++ or Python functions (image processing, DL, linear algebra, signal processing kernels) paired with input data (grayscale ImageNet images, Vicuna model weights, or random float vectors).

Output: Translated tensor operation code in a target DSL (NumPy, TensorFlow, PyTorch, MLX, TPC-C, or Gemmini) and its measured execution time on specified hardware.

Scoring recipe

def compute_metrics(predictions, gold):
    baseline_time = run_baseline(gold)
    translated_code = predictions['translated_code']
    translated_time = run_translated(translated_code)
    kernel_perf = baseline_time / translated_time
    e2e_time = translated_time + data_transfer_overhead
    e2e_perf = baseline_time / e2e_time
    return kernel_perf, e2e_perf

Common pitfalls

• Kernel performance excludes data transfer overhead, while end-to-end performance includes it; confusing the two leads to incorrect performance claims.
• Gemmini accelerator requires square matrix inputs, necessitating padding that inflates data volume and skews performance if not accounted for.
• Synthesis timeout is 1 hour; benchmarks failing to synthesize within this limit are excluded from average timing calculations.

Evidence (verbatim from paper)

Kernel performance focuses on computation time excluding data transfer overhead. We see significant improvements as illustrated in Figures 9, with an average speedup of 105.1× across all benchmarks. ... End-to-end Performance. While frameworks and accelerators deliver substantial kernel performance enhancements, a comprehensive assessment must account for end-to-end benchmark times, encompassing initial setup and data movement between the host (CPU) and the accelerator device.

Citation

@misc{qiu2024tenspiler,
  title={Tenspiler: A Verified Lifting-Based Compiler for Tensor Operations (Extended Version)},
  author={Qiu et al. (2024)},
  year={2024},
  note={arXiv:2404.18249}
}

• arXiv: 2404.18249