Datetime Eval

datetime-eval

DATETIME: A new benchmark to measure LLM translation and reasoning capabilities — Gaere et al. (2025) (arXiv:2504.16155, 2025)

What this evaluates

Evaluates large language models on their ability to parse, translate, and perform arithmetic reasoning with datetime information. It probes structured string formatting (ISO-8601) and multi-step temporal calculations across diverse linguistic contexts.

Datasets

• DATETIME — total ?; splits: test (-1); repo https://github.com/EdGaere/DATETIME

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy: the fraction of instances where the model's output exactly matches the ground truth datetime string or computed result.

Input / output format

Input: Natural language prompts requesting datetime translation, arithmetic operations (e.g., adding days), or mixed tasks requiring both parsing and computation.

Output: ISO-8601 formatted datetime strings or computed datetime values, depending on the task type.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_val in zip(predictions, gold):
        if normalize(pred) == normalize(gold_val):
            correct += 1
    return correct / len(gold)

Common pitfalls

• Strict ISO-8601 formatting requirements mean minor deviations (e.g., timezone offsets, date separators) cause exact-match failures.
• Arithmetic tasks require handling edge cases like leap years, month boundaries, and timezone conversions correctly.
• Models may hallucinate plausible but incorrect datetime strings if not strictly constrained to the ground truth format.

Evidence (verbatim from paper)

DATETIME introduces a synthetic, high-fidelity benchmark to evaluate LLMs on datetime translation and reasoning, revealing significant gaps in arithmetic and structured parsing capabilities. Tasks span translation to ISO-8601 format, datetime arithmetic (e.g., adding days), and mixed tasks requiring both, with synthetic data covering a broad temporal range and linguistic variability. Models are evaluated using accuracy.

Citation

@misc{gaere2025datetime,
  title={DATETIME: A new benchmark to measure LLM translation and reasoning capabilities},
  author={Gaere et al. (2025)},
  year={2025},
  note={arXiv:2504.16155}
}

• arXiv: 2504.16155