Tsaqa Eval

tsaqa-eval

TSAQA: Time Series Analysis Question And Answering Benchmark — Jing et al. (2026) (arXiv:2601.23204, 2026)

What this evaluates

Evaluates large language models' ability to perform time series analysis and reasoning across six tasks (anomaly detection, classification, characterization, comparison, data transformation, and temporal relationship) using three question formats (true-or-false, multiple-choice, and puzzling).

Datasets

• TSAQA — total 210000; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • The proportion of correctly answered questions out of the total number of questions, averaged over all samples rather than per-task averages.

Input / output format

Input: Time series data (often accompanied by domain context as text) paired with a question in one of three formats: true-or-false (TF), multiple-choice (MC), or puzzling (PZ) requiring chronological ordering.

Output: A single answer choice (e.g., A/B/C/D for MC, True/False for TF, or a reordered sequence for PZ).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Models exhibit a 'Smoothness Bias' on PZ questions, incorrectly predicting smoother transitions than ground truth.
• Performance generally declines as input length increases, except for the Temporal Relationship task where longer inputs help.
• Results should be averaged over all samples rather than computing per-task averages first.

Evidence (verbatim from paper)

Accuracy Correlate Analysis. Input Lengths. Figure[3] in Appendix) illustrates the relationship between input length and model accuracy. Across all six models and five tasks, excluding the Temporal Relationship task, we observe a consistent trend that performance declines as input length increases, indicating that longer inputs correspond to more difficult questions.

Citation

@misc{jing2026tsaqa,
  title={TSAQA: Time Series Analysis Question And Answering Benchmark},
  author={Jing et al. (2026)},
  year={2026},
  note={arXiv:2601.23204}
}

• arXiv: 2601.23204