Liveclin Eval

liveclin-eval

LiveClin: A Live Clinical Benchmark without Leakage — Wang et al. (2026) (arXiv:2602.16747, 2026)

What this evaluates

LiveClin probes real-world clinical reasoning and longitudinal case management by evaluating models on dynamic, multimodal patient scenarios. It tests the ability to maintain context across sequential diagnostic, treatment, and follow-up questions while resisting data contamination from static training corpora.

Datasets

• LiveClin — total ?; splits: test (-1); repo https://github.com/AQ-MedAI/LiveClin

Metrics

• Case Accuracy (primary) — range: percent
  • A case is deemed correct only if all of its sequential questions are answered correctly. It is calculated as the percentage of cases where every turn in the conversation matches the gold standard.

Input / output format

Input: Conversational, zero-shot prompts where the full conversation history is maintained as context for each subsequent question. Cases are multimodal, spanning text, images, and tables across diagnosis, treatment, and long-term management stages.

Output: Sequential answers to multi-turn questions within a clinical case conversation.

Scoring recipe

def compute_case_accuracy(predictions, gold):
    # predictions, gold: lists of model/gold answers per sequential question in a case
    all_correct = all(p == g for p, g in zip(predictions, gold))
    return 1.0 if all_correct else 0.0

Common pitfalls

• Maintaining full conversation history across sequential questions can cause context degradation or attention loss, especially in late-stage follow-up questions.
• Performance is highly modality-dependent; models excel at extracting structured data (e.g., Diagrams) but fail on complex inference tasks (e.g., Pathology, Biosignals).
• The zero-shot conversational setup does not allow for iterative refinement or feedback, penalizing models that typically rely on multi-turn clarification.

Evidence (verbatim from paper)

To faithfully simulate sequential clinical encounters, we employed a conversational, zero-shot evaluation protocol. The full conversation history is maintained as context for each subsequent question, forcing the model to continuously integrate new information. For reproducibility, we set temperature to 0 for most models, adopting official recommended configurations for those with specific reasoning modes. Our primary metric is Case Accuracy, a stringent measure where a case is deemed correct only if all of its sequential questions are answered correctly.

Citation

@misc{wang2026liveclin,
  title={LiveClin: A Live Clinical Benchmark without Leakage},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.16747}
}

• arXiv: 2602.16747