Timer Eval

timer-eval

TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records — Cui et al. (2025) (arXiv:2503.04176, 2025)

What this evaluates

This benchmark evaluates a model's ability to perform temporal reasoning and extract accurate information from longitudinal electronic health records (EHRs). It probes whether models can correctly synthesize evidence across multiple time-stamped clinical visits, adhere to specified temporal boundaries, and maintain accuracy over long patient timelines.

Datasets

• TIMER-Bench — total ?; splits: test (-1)
• MedAlign — total ?; splits: test (-1)

Metrics

• Correct (primary) — range: percent
  • LLM-as-Judge (GPT-4o-mini) scores response correctness based on clinician-validated prompts. Correlates at |ρ|=0.94 with human judgment.
• Complete — range: percent
  • LLM-as-Judge scores response completeness based on coverage of required temporal evidence. Correlates at |ρ|=0.89 with human judgment.
• BERTScore — range: [0, 1]
  • Token-level representation similarity using distilbert-based-uncased embeddings.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between generated and reference text.

Input / output format

Input: Chunked longitudinal EHR patient timeline (structured data + clinical notes) paired with a natural language instruction/query specifying a temporal scope or trend to analyze.

Output: Open-text response containing the model's analysis, trend description, or answer to the clinical query.

Scoring recipe

# LLM-as-Judge (Correctness/Completeness)
judge = GPT4oMini()
correctness_score = judge.evaluate(prediction, gold, prompt_template="Appendix F")
completeness_score = judge.evaluate(prediction, gold, prompt_template="Appendix F")

# Automated Metrics
bertscore = compute_bertscore(prediction, gold, model="distilbert-base-uncased")
rouge_l = compute_rouge_l(prediction, gold)

# Aggregate (reported as mean ± std over 10,000 bootstrap resamples)
metrics = {
    "Correct": correctness_score,
    "Complete": completeness_score,
    "BERTScore": bertscore,
    "ROUGE-L": rouge_l
}

Common pitfalls

• Truncating long EHRs to fit context windows often removes recent records, severely hurting recency-focused tasks like MedAlign.
• Synthetic data generation exhibits a 'lost-in-the-middle' bias, causing models to default to extreme time points (start/end of timeline) rather than mid-timeline events.
• LLM-as-Judge requires specific prompting (Appendix F) and validation against clinicians; using default prompts may misalign with the reported correlation metrics.

Evidence (verbatim from paper)

We evaluate models’ open-text responses using LLM-Judge that assesses response correctness and completeness... We also employ head-to-head comparisons and automated metrics derived from token-level representations, including BertScore (using distilbert-based-uncased), ROUGE-L, CHRF, and METEOR to provide standard assessment of response quality. All LLM-based evaluations use GPT-4o-mini as the judge.

Citation

@misc{cui2025timer,
  title={TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records},
  author={Cui et al. (2025)},
  year={2025},
  note={arXiv:2503.04176}
}

• arXiv: 2503.04176