Wasm Bench Eval

wasm-bench-eval

A fast in-place interpreter for WebAssembly — Titzer (2022) (arXiv:2205.01183, 2022)

What this evaluates

Evaluates the performance trade-offs of a WebAssembly in-place interpreter against state-of-the-art Wasm engines across translation time, translation space overhead, and execution time on a standard benchmark suite.

Datasets

• PolyBenchC-4.2.1 MEDIUM — total 24; splits: test (24)

Metrics

• translation_time (primary) — range: seconds/byte
  • Average translation time (seconds) divided by the number of input bytes translated. Normalizes for benchmark size and tiering strategy.
• translation_space — range: bytes/byte
  • Memory overhead of translation measured as output bytes generated per input byte of code.
• execution_time — range: seconds
  • Absolute execution time in seconds for the complete process, often normalized to a baseline engine (e.g., v8-turbofan or wasm3).

Input / output format

Input: WebAssembly bytecode modules from the PolyBenchC-4.2.1 MEDIUM dataset.

Output: Performance measurements (translation time, translation space, execution time) averaged over 100 runs per engine configuration.

Scoring recipe

def compute_metrics(measurements, input_bytes, baseline_time):
    # measurements: list of dicts with 'trans_time', 'trans_space', 'exec_time' for 100 runs
    avg_trans_time = sum(m['trans_time'] for m in measurements) / len(measurements)
    avg_trans_space = sum(m['trans_space'] for m in measurements) / len(measurements)
    avg_exec_time = sum(m['exec_time'] for m in measurements) / len(measurements)
    
    trans_time_ratio = avg_trans_time / input_bytes
    trans_space_ratio = avg_trans_space / input_bytes
    exec_time_ratio = avg_exec_time / baseline_time
    
    return {
        'translation_time': trans_time_ratio,
        'translation_space': trans_space_ratio,
        'execution_time': avg_exec_time,
        'execution_time_ratio': exec_time_ratio
    }

Common pitfalls

• Normalization to input bytes is required to account for varying benchmark sizes and tiering strategies (lazy vs eager).
• Execution time includes full OS process startup/teardown, heavily skewing results for short-running benchmarks.
• Translation space only measures generated code bytes, excluding debugging metadata, thus underestimating real memory overhead.

Evidence (verbatim from paper)

We plot the average translation time, divided by the number of input bytes translated. The ratio of translation time to input bytes normalizes differences in tiering strategy (e.g. lazy compilation) and benchmark size.

Citation

@misc{titzer2022fastinplaceinterpreter,
  title={A fast in-place interpreter for WebAssembly},
  author={Titzer (2022)},
  year={2022},
  note={arXiv:2205.01183}
}

• arXiv: 2205.01183