# Acol Interpreter Benchmark Eval

> Evaluates the execution speed and dispatching efficiency of different interpreter implementations (AST vs. bytecode variants) for a simple imperative language (ACOL) in Prolog. Use when the user wants to benchmark on ACOL Interpreter Benchmarks, or asks about evaluating this task. Reports geometric_mean_runtime.

- Skill: `qhjqhj00/acol-interpreter-benchmark-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/acol-interpreter-benchmark-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/acol-interpreter-benchmark-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/acol-interpreter-benchmark-eval

---


# acol-interpreter-benchmark-eval

> On the Performance of Bytecode Interpreters in Prolog — Korner et al. (2020) (arXiv:2008.12543, 2020)

## What this evaluates

Evaluates the execution speed and dispatching efficiency of different interpreter implementations (AST vs. bytecode variants) for a simple imperative language (ACOL) in Prolog.

## Datasets

- **ACOL Interpreter Benchmarks** — total 6; splits: test (6); repo https://github.com/pkoerner/prolog-interpreters

## Metrics

- `geometric_mean_runtime` **(primary)** — range: other
  - Geometric mean of 10 execution runs per interpreter per benchmark. Also reported as normalized runtime relative to the baseline AST interpreter.

## Input / output format

**Input**: ACOL programs (while-loops, if-statements, assignments, comparisons, additions/subtractions) with pre-initialized environment variables.

**Output**: Execution time in seconds (mean ± 0.95 confidence interval over 10 runs).

## Scoring recipe

```python
runtimes = []
for _ in range(10):
    t = run_program_in_interpreter(program, interpreter)
    runtimes.append(t)
mean_runtime = geometric_mean(runtimes)
normalized_runtime = mean_runtime / baseline_ast_mean_runtime
return mean_runtime, normalized_runtime
```

## Common pitfalls

- Compilation time is explicitly excluded; only interpreter dispatch/runtime is measured.
- Geometric mean is used for aggregation instead of arithmetic mean, as specified by the authors.
- Small handwritten programs may benefit from AST caching, skewing results compared to larger generated programs.

## Evidence (verbatim from paper)

> Each program was executed with every interpreter ten times. The runtime consists only of the time spent in the interpreter. Compilation time is excluded, as it is not implemented efficiently and, ultimately, not relevant. The mean value is determined by the geometric mean as proposed by [5].

## Citation

```bibtex
@misc{korner2020bytecode,
  title={On the Performance of Bytecode Interpreters in Prolog},
  author={Korner et al. (2020)},
  year={2020},
  note={arXiv:2008.12543}
}
```

- arXiv: 2008.12543