Time Eval

time-eval

TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios — Wei et al. (2025) (arXiv:2505.12891, 2025)

What this evaluates

Evaluates large language models' ability to reason about temporal information in real-world scenarios. It probes capabilities ranging from basic time retrieval and localization to complex event ordering, duration comparison, and counterfactual temporal reasoning across knowledge-intensive, dynamic news, and long-form dialogue contexts.

Datasets

• TimE-Wiki — total ?; splits: test (-1); repo https://github.com/sylvain-wei/TIME
• TimE-News — total ?; splits: test (-1); repo https://github.com/sylvain-wei/TIME
• TimE-Dial — total ?; splits: test (-1); repo https://github.com/sylvain-wei/TIME

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • 1 if the predicted answer exactly matches the ground truth string, 0 otherwise.
• F1 score — range: [0, 1]
  • Token-level F1 score computed between the predicted free-form answer and the ground truth.
• Option-level F1 (macro F1) — range: [0, 1]
  • Macro-averaged F1 score across all options for single-choice and multiple-choice QA tasks.

Input / output format

Input: Contextual passages (encyclopedic knowledge, news articles, or multi-turn dialogues) paired with questions requiring temporal understanding, formatted as free-form or multiple-choice prompts.

Output: Free-form text responses or selected option letters corresponding to the question.

Scoring recipe

def compute_metrics(predictions, golds, task_types):
    em_scores, f1_scores = [], []
    for pred, gold, task in zip(predictions, golds, task_types):
        if task == 'timeline':
            em_scores.append(1.0 if pred.strip() == gold.strip() else 0.0)
        else:
            f1_scores.append(token_f1(gold, pred))
    return {'EM': mean(em_scores), 'F1': mean(f1_scores)}

Common pitfalls

• The Extract task is explicitly excluded from evaluations using the RAG framework on TimE-News and TimE-Lite-News.
• Retriever strategy (BM25, Vector, Hybrid) heavily influences TimE-News results, sometimes dominating over model architecture differences.
• The Timeline task requires ordering three events and consistently yields very low accuracy (<30%) across all models, making it a poor standalone metric for basic temporal understanding.

Evidence (verbatim from paper)

We evaluate free-form QA tasks with token-level metrics: Exact Match (EM) for the Timeline task, and F1 score for other free-form QA tasks. For single-choice and multiple-choice QA tasks, we use option-level F1 scores, emphasizing macro F1 for a comprehensive evaluation across all options.

Citation

@misc{wei2025time,
  title={TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2505.12891}
}

• arXiv: 2505.12891