Temporalbench Eval

temporalbench-eval

TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks — Weng et al. (2026) (arXiv:2602.13272, 2026)

What this evaluates

TemporalBench probes LLM-based agents' ability to perform contextual and event-informed temporal reasoning across four distinct task families. It disentangles historical pattern interpretation, context-free forecasting, contextual alignment, and event-conditioned adaptation to reveal whether numerical prediction accuracy correlates with qualitative temporal judgment.

Datasets

• FreshRetailNet — total ?; splits: test (-1)
• PSML — total ?; splits: test (-1)
• Causal Chambers — total ?; splits: test (-1)
• MIMIC — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct discrete choices among valid instances. Answers labeled Uncertain or Inconclusive are excluded from the denominator to avoid penalizing models on ambiguous cases.
• MAE — range: [0, inf)
  • Mean Absolute Error: the average of absolute differences between predicted and ground-truth future values over the forecast horizon.
• sMAPE — range: [0, inf)
  • Symmetric Mean Absolute Percentage Error: 100 * mean(2 * |pred - gold| / (|pred| + |gold|)), used for single or low-dimensional series.
• OW_sMAPE — range: [0, inf)
  • Overall Weighted sMAPE: a scale-invariant aggregation of sMAPE across multiple correlated clinical time series to prevent high-variance signals from dominating.
• OW_RMSSE — range: [0, inf)
  • Overall Weighted Root Mean Squared Scaled Error: a scaled error metric aggregated across multiple series to enable fair comparison across heterogeneous clinical scales.

Input / output format

Input: Historical time-series values, contextual narratives or event descriptions, and task-specific prompts (multiple-choice questions or direct forecasting requests).

Output: For reasoning tasks: discrete multiple-choice labels (or 'Uncertain'/'Inconclusive'). For forecasting tasks: numerical predictions for the specified future horizon.

Scoring recipe

def score(predictions, gold, task_type, dataset_type):
    if task_type == 'mcq':
        valid_mask = [p not in ['Uncertain', 'Inconclusive'] for p in predictions]
        valid_preds = [p for p, m in zip(predictions, valid_mask) if m]
        valid_gold = [g for g, m in zip(gold, valid_mask) if m]
        return sum(p == g for p, g in zip(valid_preds, valid_gold)) / len(valid_preds)
    elif task_type == 'forecasting':
        if dataset_type == 'MIMIC':
            return ow_smape(predictions, gold), ow_rmsse(predictions, gold)
        else:
            return mae(predictions, gold), smape(predictions, gold)

Common pitfalls

• Uncertain or Inconclusive model outputs are explicitly excluded from accuracy computation; failing to filter them will artificially deflate scores.
• High numerical forecasting accuracy (low MAE/sMAPE) does not correlate with high qualitative reasoning accuracy, so evaluating only forecasting metrics misses the benchmark's core diagnostic purpose.
• Multi-series datasets like MIMIC require overall weighted metrics (OW_sMAPE/OW_RMSSE) to avoid scale domination; applying standard MAE/sMAPE directly will produce misleading results.

Evidence (verbatim from paper)

For multiple-choice question-answering tasks, including T1, T3, and the qualitative components of T2 and T4, we report accuracy as the primary metric. Accuracy is chosen to emphasize unambiguous decision correctness... For numerical forecasting tasks in T2 and T4, we evaluate prediction quality using error-based metrics computed between predicted and ground-truth future values over the valid forecast horizon. For the MIMIC dataset, forecasting involves multiple correlated clinical time series with heterogeneous scales. To ensure fair aggregation across series and avoid domination by high-variance signals, we use overall weighted metrics, specifically OW_sMAPE and OW_RMSSE.

Citation

@misc{weng2026temporalbench,
  title={TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks},
  author={Weng et al. (2026)},
  year={2026},
  note={arXiv:2602.13272}
}

• arXiv: 2602.13272