Event Inference Eval

event-inference-eval

Inferring Events from Time Series using Language Models — Mingtian Tan et al. (2025) (arXiv:2503.14190, 2025)

What this evaluates

Evaluates large language models' ability to infer natural language event sequences from real-valued time series data (specifically win probabilities in sports). It probes causal reasoning, temporal context understanding, and the model's capacity to distinguish underlying time series dynamics from linguistic descriptions.

Datasets

• NBA & NFL Event Inference Benchmark — total 200; splits: test (200); repo https://github.com/BennyTMT/GAMETime

Metrics

• accuracy (primary) — range: percent
  • Calculated as the number of correctly inferred event sequences divided by the total number of questions (200).

Input / output format

Input: A segment of real-valued time series data (Team A's win probabilities), optionally augmented with timestamps, scores, partial events, or team/player names depending on the experimental condition. Multiple-choice options include the ground-truth event sequence and negative sequences sampled from other games.

Output: A natural language sequence of events corresponding to the provided time series segment.

Scoring recipe

def compute_accuracy(predictions, gold, total=200):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / total
# Note: 'Invalid' outputs (failure to return a valid sequence) are counted as incorrect.

Common pitfalls

• Models may rely on memorized team/player names or numerical values in event descriptions rather than true time-series reasoning; the benchmark mitigates this by using generic labels and post-cutoff games.
• Chain-of-Thought prompting improves reasoning but increases the rate of format-invalid outputs, which must be strictly filtered or counted as incorrect.
• Performance varies significantly across domains (e.g., sports vs. crypto/health), so results from one domain do not generalize to others without re-evaluation.

Evidence (verbatim from paper)

GPT-4o achieves an accuracy of 41%, and DS-R1-distill-Qwen-32B reaches 68%, while o1 performs the best, with an accuracy of 83%.

Citation

@misc{tan2025inferring,
  title={Inferring Events from Time Series using Language Models},
  author={Mingtian Tan et al. (2025)},
  year={2025},
  note={arXiv:2503.14190}
}

• arXiv: 2503.14190