Traveler Temporal Reasoning Eval

traveler-temporal-reasoning-eval

TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References — Deigmöller et al. (2025) (arXiv:2505.01325, 2025)

What this evaluates

This benchmark evaluates large language models' ability to perform event-temporal reasoning by resolving explicit, implicit, and vague temporal references across synthetic household event chains. It systematically probes how model performance degrades with increasing event set length and varying levels of temporal explicitness.

Datasets

• TRAVELER — total 3300; splits: test (3300)

Metrics

• accuracy (primary) — range: [0, 1]
  • For explicit and implicit temporal references, accuracy is computed as binary exact-match accuracy between the model's response and the automatically derived ground truth. For vague temporal references, accuracy is computed probabilistically by weighting the model's response against survey-derived likelihoods that a given adverbial applies to an event at a specific time delta.

Input / output format

Input: A list of synthetic events formatted as tuples or JSON objects containing Event Type, Subject, Location, and Timestamp, followed by a natural language question template instantiated with a temporal expression (e.g., 'Did {subject} {event_type} {location} {temporal_expression}?').

Output: String (subject name or 'nobody'), Boolean (yes/no), Integer (count), or Date, depending on the question template.

Scoring recipe

def score(predictions, golds, event_sets, vague_survey_data, ref_time):
    scores = []
    for pred, gold, events, question in zip(predictions, golds, event_sets, questions):
        if question.category in ['explicit', 'implicit']:
            scores.append(1.0 if pred == gold else 0.0)
        else: # vague
            matching_events = [e for e in events if match_criteria(e, question)]
            prob_at_least_one = 1.0
            for e in matching_events:
                dt = ref_time - e.timestamp
                p = vague_survey_data[e.type][question.adverbial][dt]
                prob_at_least_one *= (1.0 - p)
            prob_at_least_one = 1.0 - prob_at_least_one
            if pred == 'yes':
                scores.append(prob_at_least_one)
            else:
                scores.append(1.0 - prob_at_least_one)
    return sum(scores) / len(scores)

Common pitfalls

• Vague temporal references do not have absolute true/false ground truths; evaluators must use the survey-derived probabilistic scoring scheme rather than binary exact match.
• The reference time for implicit and vague expressions must be explicitly fixed (e.g., 'Today is 2023-09-29 22:18') to compute time deltas consistently.
• Performance heavily depends on event set length; benchmarks using only small sets (e.g., 5 events) will overestimate model capabilities compared to the full 100-event scale.

Evidence (verbatim from paper)

For the evaluation, we utilize two strategies. First, we assess how well LLMs identify events based on explicit and implicit relative to speech time temporal references, using a binary classification approach. Second, we evaluate vague temporal expressions probabilistically, as no absolute true-or-false decisions can be made. The LLMs response is evaluated using binary accuracy by comparing it’s response with the correct answer.

Citation

@misc{deigmoeller2025traveler,
  title={TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References},
  author={Deigmöller et al. (2025)},
  year={2025},
  note={arXiv:2505.01325}
}

• arXiv: 2505.01325