Linglanmidian Eval

linglanmidian-eval

LingLanMiDian: Systematic Evaluation of LLMs on TCM Knowledge and Clinical Reasoning — Hua et al. (2026) (arXiv:2602.01779, 2026)

What this evaluates

Evaluates LLMs on Traditional Chinese Medicine (TCM) knowledge recall, multi-hop clinical reasoning, information extraction, and clinical decision-making. It probes synonym-tolerant clinical labeling, robustness on curated hard subsets, and performance across diverse TCM-specific task formats including QA, NER, and dosage prediction.

Datasets

• LingLanMiDian — total ?; splits: test (-1), test-hard (400); repo https://github.com/TCMAI-BJTU/LingLan

Metrics

• Accuracy (primary) — range: [0, 1] or percent
  • Fraction of correctly predicted options or decisions. Calculated as correct predictions divided by total instances.
• F1 — range: [0, 1] or percent
  • Harmonic mean of precision and recall. Macro-averaged across classes or entities within each subtask.
• Precision — range: [0, 1] or percent
  • Ratio of correctly predicted positive instances to all predicted positive instances. Macro-averaged within subtasks.
• Recall — range: [0, 1] or percent
  • Ratio of correctly predicted positive instances to all actual positive instances. Macro-averaged within subtasks.
• char-F1 — range: [0, 1] or percent
  • Character-level F1 score used for cloze completion tasks, measuring overlap between predicted and gold character sequences.
• MAE — range: other
  • Mean Absolute Error for dosage estimation tasks, measuring average absolute difference between predicted and gold dosage values.
• Cosine similarity — range: other
  • Cosine similarity between predicted and gold dosage vectors, measuring directional alignment of dosage predictions.

Input / output format

Input: Multiple-choice, single-choice, and cloze questions for knowledge tasks; raw clinical EMR and classical TCM texts for NER; patient case descriptions for syndrome, treatment, prescription, and dosage prediction; and decision recognition prompts reformulating clinical problems into single-choice tasks.

Output: Selected option indices for QA/DR tasks; extracted entity spans for NER; predicted syndrome/treatment/prescription/dosage values for reasoning tasks; and chosen decision option for DR tasks.

Scoring recipe

def compute_linglan_metrics(predictions, golds, task_type):
    if task_type in ['QA', 'DR']:
        return accuracy_score(golds, predictions)
    elif task_type in ['NER', 'DTR']:
        return f1_score(golds, predictions, average='macro')
    elif task_type == 'Dosage':
        return mae(golds, predictions), cosine_similarity(golds, predictions)
    # 1. Macro-average all metrics within each subtask
    # 2. Average subtask scores to get per-task-type score
    # 3. Final overall average = mean(per-task-type scores)
    # Note: MAE is excluded from the final overall average calculation.

Common pitfalls

• Strict exact matching (DTR) yields significantly lower scores than the synonym-tolerant labeling protocol (DTR-F1); evaluators must use the synonym-tolerant variant for clinical reasoning tasks to avoid underestimating model capability.
• The overall average is not a simple global mean; it is computed by first averaging metrics within subtasks, then averaging across task types equally, which can mask performance variations within specific domains.
• Hard subsets cause a consistent ~20-point drop in scores across all models; evaluating only on the full test set overestimates real-world clinical reasoning capability and robustness.

Evidence (verbatim from paper)

Under a unified zero-shot protocol, with identical decoding settings where supported, we evaluate 14 models on 13 subtasks spanning five domains. Results are reported for both the full test sets and the curated 400-item hard subsets, and all scores are macro-averaged within each subtask. The overall average for each model is obtained by first averaging over sub-tasks and metrics (accuracy, F1, cosine, precision, recall) within each task type, then taking the mean of these per-task-type scores so that each task type is weighted equally.

Citation

@misc{hua2026linglanmidian,
  title={LingLanMiDian: Systematic Evaluation of LLMs on TCM Knowledge and Clinical Reasoning},
  author={Hua et al. (2026)},
  year={2026},
  note={arXiv:2602.01779}
}

• arXiv: 2602.01779