qhjqhj00 tcm_best4sdt-eval
A benchmark dataset for evaluating Syndrome Differentiation and Treatment in large language models — Li et al. (2025) (arXiv:2512.02816, 2025)
What this evaluates
This benchmark evaluates large language models' capabilities in Traditional Chinese Medicine (TCM) clinical reasoning, specifically focusing on syndrome differentiation and treatment decision-making. It probes the model's ability to accurately diagnose pathological patterns, formulate appropriate herbal prescriptions, and adhere to medical ethics and safety guidelines across 27 dimensions.
Datasets
- TCM-BEST4SDT — total 600; splits: test (600); repo https://github.com/DYJG-research/TCM-BEST4SDT
Metrics
selected-response evaluation(primary) — range: percent- Accuracy calculated by comparing the model's selected option(s) against the expert-annotated gold standard. Scores are adjusted based on whether the question is single-selection or multiple-selection, with multiple-choice items using sophisticated distractors to increase discriminative power.
reward model evaluation— range: [0, 1]- A specialized reward model outputs a continuous score that objectively quantifies prescription-syndrome congruence, measuring how well the generated treatment aligns with the diagnosed pathological pattern.
CoT Content Completeness— range: percent- Measures the coverage of key patient information present within the model's Chain-of-Thought reasoning trace.
CoT Accuracy— range: percent- Assesses the consistency of elements cited in the Chain-of-Thought with the original clinical case to identify hallucinations or reasoning deviations.
Input / output format
Input: Text-based clinical cases, classical case records, or examination questions. Inputs are anonymized, deduplicated, and cleaned. Questions are presented as multiple-choice (single or multiple selection) or open-ended prompts requiring diagnostic reasoning and prescription generation.
Output: For multiple-choice questions: the model's selected option letter(s). For open-ended questions: generated text containing diagnostic reasoning, herbal prescriptions, treatment principles, and safety/ethics compliance statements.
Scoring recipe
def score_selected_response(pred, gold, q_type):
if q_type == 'single':
return 1.0 if pred.strip() == gold.strip() else 0.0
elif q_type == 'multiple':
pred_set = set(pred.split(','))
gold_set = set(gold.split(','))
return len(pred_set & gold_set) / len(gold_set) if gold_set else 0.0
def score_reward_model(pred_prescription, gold_syndrome):
# Uses specialized reward model to output congruence score
return reward_model.predict(congruence=(pred_prescription, gold_syndrome))
def score_cot_metrics(pred_cot, gold_case):
completeness = coverage_score(pred_cot, key_patient_info)
accuracy = consistency_score(pred_cot, gold_case)
return completeness, accuracy
Common pitfalls
- Failing to randomize option order during selected-response evaluation, which can artificially inflate accuracy due to position bias.
- Overlooking process-oriented metrics (CoT Content Completeness and CoT Accuracy) and only evaluating final diagnostic/treatment outcomes, missing hallucination or reasoning deviation detection.
- Using generic LLM judges without expert-designed prompts, leading to inconsistent or clinically invalid scoring for TCM-specific nuances.
Evidence (verbatim from paper)
The first is selected-response evaluation. This method involves multiple rounds of independent evaluation for each question, randomizes the option order, and calculates scores using different scoring strategies based on the question type. The second is judge model evaluation. This mechanism utilizes an LLM, combined with expert-designed prompts, to score the responses of the model under evaluation. The third is reward model evaluation. This study developed a specialized reward model to objectively quantify prescription-syndrome congruence.
Citation
@misc{li2025tcm_best4sdt,
title={A benchmark dataset for evaluating Syndrome Differentiation and Treatment in large language models},
author={Li et al. (2025)},
year={2025},
note={arXiv:2512.02816}
}
- arXiv: 2512.02816