Interpreter Bench Eval

interpreter-bench-eval

Speculative Staging for Interpreter Optimization — Brunthaler (2013) (arXiv:1310.2300, 2013)

What this evaluates

Evaluates the runtime performance and memory efficiency of a speculatively staged Python interpreter against standard baselines like CPython and PyPy. It probes the interpreter's ability to eliminate dynamic type-checking overhead and optimize instruction dispatch through compile-time specialization.

Datasets

• Computer Language Benchmarks Game — total ?; splits: test (6)

Metrics

• speedup (primary) — range: other
  • Speedup is calculated as the execution time of the baseline interpreter (CPython 3.2.3 with switch dispatch) divided by the execution time of the optimized interpreter. Results are reported as the geometric mean across all benchmarks.

Input / output format

Input: Python benchmark programs (binarytrees, fannkuch, fasta, mandelbrot, nbody, spectralnorm) executed under different interpreter configurations.

Output: Execution time per run and normalized speedup factor relative to the CPython 3.2.3 baseline.

Scoring recipe

def geometric_mean(values):
    n = len(values)
    product = 1.0
    for v in values:
        product *= v
    return product ** (1.0 / n)

# For each benchmark and interpreter pairing:
rep_speedups = [baseline_time / opt_time for _ in range(30)]
bench_speedup = geometric_mean(rep_speedups)

# Final metric: geometric mean across all 6 benchmarks
final_metric = geometric_mean([bench_speedup for bench in benchmarks])

Common pitfalls

• Startup and warm-up times are included in PyPy's measured times, which can skew JIT comparison fairness.
• Benchmarks are originally written for Python 2; minor modifications were needed for Python 3 compatibility (e.g., fannkuch), potentially affecting direct comparison.
• Memory management differences (reference counting vs. generational GC) mean memory usage metrics do not directly correlate with execution speed.

Evidence (verbatim from paper)

we use 30 repetitions for each pairing of a benchmark with an interpreter to get stable results; we report the geometric mean of these repetitions, thereby minimizing bias towards outliers. Table 2 lists the geometric mean of speedups per benchmark that we measured on our Intel Nehalem system.

Citation

@misc{brunthaler2013speculative,
  title={Speculative Staging for Interpreter Optimization},
  author={Brunthaler (2013)},
  year={2013},
  note={arXiv:1310.2300}
}

• arXiv: 1310.2300