Explaincpe Eval

explaincpe-eval

ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination — Dongfang Li et al. (2023) (arXiv:2305.12945, 2023)

What this evaluates

This benchmark evaluates large language models on Chinese medical multiple-choice questions, specifically probing their ability to select correct answers and generate faithful, logically consistent free-text explanations. It measures both factual accuracy and the quality of interpretability in high-stakes healthcare domains.

Datasets

• ExplainCPE — total ?; splits: train (-1), test (-1); repo https://github.com/HITsz-TMG/ExplainCPE

Metrics

• Accuracy (primary) — range: percent
  • Percentage of test instances where the model's predicted answer option exactly matches the gold standard option.
• ROUGE-1 — range: [0, 1]
  • Unigram-level recall/precision F1 score computed between the generated explanation and the gold explanation.
• ROUGE-2 — range: [0, 1]
  • Bigram-level recall/precision F1 score computed between the generated explanation and the gold explanation.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence-based recall/precision F1 score computed between the generated explanation and the gold explanation.

Input / output format

Input: A multiple-choice medical question with options, optionally preceded by few-shot examples (zero, one, four, or eight). Prompts may include or exclude explicit instructions.

Output: The selected answer option followed by a free-text explanation justifying the choice.

Scoring recipe

def score(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p['option'] == g['option'])
    acc = (correct / len(predictions)) * 100
    r1, r2, rl = 0.0, 0.0, 0.0
    for p, g in zip(predictions, golds):
        r1 += rouge1_score(g['explanation'], p['explanation'])
        r2 += rouge2_score(g['explanation'], p['explanation'])
        rl += rougeL_score(g['explanation'], p['explanation'])
    n = len(predictions)
    return {'Accuracy': acc, 'ROUGE-1': r1/n, 'ROUGE-2': r2/n, 'ROUGE-L': rl/n}

Common pitfalls

• ROUGE metrics only measure lexical overlap and do not capture factual correctness, logical consistency, or hallucination of contraindications in medical explanations.
• Few-shot examples for 4-shot and 8-shot settings were manually selected to balance question types, which may introduce selection bias and limit reproducibility compared to random sampling.
• Prompt variations (with/without instructions) interact differently with chat vs. non-chat models, making direct cross-model comparisons highly sensitive to prompt formatting.

Evidence (verbatim from paper)

we designed prompts to request the model to provide an answer option along with an explanation in the test set. Table 3: Performance comparison on ExplainCPE dataset. ModelAcc(%)Rouge-1Rouge-2Rouge-L

Citation

@misc{li2023explaincpe,
  title={ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination},
  author={Dongfang Li et al. (2023)},
  year={2023},
  note={arXiv:2305.12945}
}

• arXiv: 2305.12945