Tempobench Eval

tempobench-eval

Mechanics of Learned Reasoning 1: TempoBench, A Benchmark for Interpretable Deconstruction of Reasoning System Performance — Holzer et al. (2025) (arXiv:2510.27544, 2025)

What this evaluates

Evaluates large language models' temporal reasoning capabilities by decomposing performance into trace-based (TTE) and causal (TCE) components. It measures how well models handle structured logical specifications with varying complexity, isolating structural factors like horizon depth and information density.

Datasets

• TempoBench — total 800; splits: TTE (400), TCE (400); repo https://github.com/nik-hz/tempobench

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • Calculates the proportion of instances where the model's generated JSON output exactly matches the ground truth specification. Evaluated separately for TTE and TCE tasks.

Input / output format

Input: JSON objects describing the atomic propositions of a system, formatted as a dictionary mapping system identifiers to lists of constraints for different states.

Output: JSON objects containing exact sentences within the specified structure, matching the ground truth format.

Scoring recipe

correct = 0
total = len(predictions)
for pred, gold in zip(predictions, golds):
    if pred == gold:  # Exact match of JSON structure and content
        correct += 1
return correct / total

Common pitfalls

• Models may struggle with exact JSON formatting, though the authors report no difficulties in practice.
• Performance drops significantly on high-complexity variants despite larger model sizes, indicating negative scaling with problem complexity.
• One-shot prompting with CoT examples is used, which may not reflect zero-shot or few-shot capabilities.

Evidence (verbatim from paper)

Testing the models on $800$ samples, $400$ for each of the TTE and TCE tasks, we evaluate each model using one-shot prompting. We include an example of a CoT solution strategy in the prompt and evaluate the models using the evaluation metrics detailed in Section [3]. Our inputs take the form of JSON objects shown in Listing [7]. During evaluation, LLMs must produce exact sentences within the JSON object.

Citation

@misc{holzer2025tempobench,
  title={Mechanics of Learned Reasoning 1: TempoBench, A Benchmark for Interpretable Deconstruction of Reasoning System Performance},
  author={Holzer et al. (2025)},
  year={2025},
  note={arXiv:2510.27544}
}

• arXiv: 2510.27544