Tsaia Eval

tsaia-eval

When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference — Wen Ye et al. (2025) (arXiv:2509.01822, 2025)

What this evaluates

Evaluates LLMs' ability to perform multi-step, constraint-aware reasoning and inference on real-world time series data. It probes compositional reasoning, numerical precision, and the capacity to assemble complex analytical or forecasting workflows via executable code generation.

Datasets

• TSAIA — total ?; splits: test (-1); repo https://github.com/USC-Melady/TSAIA

Metrics

• Success Rate (primary) — range: [0, 1]
  • Proportion of task instances where the model's generated code and final output satisfy predefined success criteria.
• MAPE — range: [0, 1]
  • Mean Absolute Percentage Error, used for forecasting tasks to measure numerical prediction accuracy.
• Accuracy — range: [0, 1]
  • Standard classification accuracy, used for tasks like trend prediction.
• Abs Error — range: [0, inf)
  • Absolute error between predicted and ground truth values, used for risk/return estimation tasks.

Input / output format

Input: Task instruction string plus a serialized time series dataset in .pkl format.

Output: Executable Python code generated by the LLM, executed within a controlled Jupyter notebook interpreter via the CodeAct agent framework.

Scoring recipe

success_count = 0
for instance in dataset:
    code = model.generate(instruction=instance.prompt, data=instance.pkl_data)
    result = execute_in_jupyter(code)
    if evaluator.check_success(instance, result):
        success_count += 1
return success_count / len(dataset)

Common pitfalls

• Models often fail to maintain numerical precision or produce correctly shaped outputs when processing structured numerical inputs.
• LLMs struggle to autonomously assemble complex multi-step workflows, such as using reference samples to calibrate detection thresholds.
• Performance is heavily biased by domain familiarity; models perform well on common metrics (e.g., Sharpe ratio) but poorly on less conventional ones (e.g., Calmar ratio).

Evidence (verbatim from paper)

The primary metric is Success Rate which is defined as the proportion of task instances for which the model output satisfies the predefined success criteria (see Table 1). For outputs deemed successful, we further evaluate quality using task-specific metrics (e.g., MAPE for forecasting, F1-score for anomaly detection), providing a more fine-grained comparison of inference quality.

Citation

@misc{ye2025whenllmmeetstimeseries,
  title={When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference},
  author={Wen Ye et al. (2025)},
  year={2025},
  note={arXiv:2509.01822}
}

• arXiv: 2509.01822