Timeseries Exam Eval

timeseries-exam-eval

TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale — Gwiazda et al. (2026) (arXiv:2604.10291, 2026)

What this evaluates

This benchmark evaluates vision-language models' ability to reason over time series data across medical, financial, and meteorological domains. It probes pattern recognition, anomaly detection, and causal reasoning using synthetically generated multiple-choice questions derived from real-world datasets.

Datasets

• PTB-XL — total 151; splits: test (151)
• MIT-BIH — total 197; splits: test (197)
• MIMIC-IV Waveform — total 205; splits: test (205)
• Yahoo Finance — total 209; splits: test (209)
• WeatherBench 2 — total 95; splits: test (95)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered multiple-choice questions out of the total number of questions. Calculated as correct predictions divided by total predictions.

Input / output format

Input: Time series visualization (image) or text representation paired with a multiple-choice question.

Output: A single letter corresponding to the chosen multiple-choice option (e.g., A, B, C, D), or a structured response containing the option.

Scoring recipe

def compute_accuracy(predictions, gold_answers, parsable=False):
    if parsable:
        valid = [(p, g) for p, g in zip(predictions, gold_answers) if is_valid_format(p)]
        return sum(1 for p, g in valid if p == g) / len(valid)
    return sum(1 for p, g in zip(predictions, gold_answers) if p == g) / len(gold_answers)

Common pitfalls

• Confusing the source datasets (e.g., PTB-XL, MIT-BIH) with the actual benchmark, which consists of synthetically generated MCQs rather than raw time series classification tasks.
• Failing to distinguish between 'General' accuracy (strict formatting required) and 'Parsable' accuracy (only valid formats counted), leading to inconsistent performance reporting.
• Assuming the evaluation measures raw time series forecasting; it actually tests reasoning over generated questions about the data.

Evidence (verbatim from paper)

In total, we have 209 samples for YFinance, 197 samples for MIT-BIH, 151 samples for PTB-XL, 205 samples for MIMIC-IV Waveform, and 95 samples for WeatherBench 2. ... Even the strongest model achieves only 51.5% average accuracy, which highlights the limitations of current LLMs.

Citation

@misc{gwiazda2026timeseriesexamagent,
  title={TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale},
  author={Gwiazda et al. (2026)},
  year={2026},
  note={arXiv:2604.10291}
}

• arXiv: 2604.10291