Huatuo 26m Eval

huatuo-26m-eval

Huatuo-26M, a Large-scale Chinese Medical QA Dataset — Jianquan Li et al. (2023) (arXiv:2305.01526, 2023)

What this evaluates

Evaluates Chinese medical question-answering capabilities through retrieval and generation tasks. It probes domain-specific knowledge retrieval from large pools and tests generative models on producing accurate, long-form medical answers.

Datasets

• Huatuo-26M — total 26000000; splits: train (-1), test (-1); repo https://github.com/FreedomIntelligence/Huatuo-26M

Metrics

• Recall@5 (primary) — range: [0, 1]
  • Measures the percentage of queries for which the correct answer appears in the top 5 retrieved passages from the candidate pool.
• MRR@10 — range: [0, 1]
  • Calculates the average of the inverse of the rank at which the first relevant document is retrieved, considering only the top 10 results.
• BLEU-4 — range: [0, 1]
  • Computes the 4-gram overlap between the generated answer and the reference answer, with a brevity penalty.
• ROUGE-L — range: [0, 1]
  • Measures the longest common subsequence of words between the generated and reference answers.

Input / output format

Input: A Chinese medical question.

Output: For retrieval: a ranked list of candidate answers. For generation: a text string representing the predicted medical answer.

Scoring recipe

def compute_recall_mrr(retrieved_lists, gold_answers, k=5):
    correct = 0
    mrr_sum = 0.0
    for preds, gold in zip(retrieved_lists, gold_answers):
        if gold in preds[:k]:
            correct += 1
        for rank, pred in enumerate(preds[:10], 1):
            if pred == gold:
                mrr_sum += 1.0 / rank
                break
    recall_k = correct / len(gold_answers)
    mrr_10 = mrr_sum / len(gold_answers)
    return recall_k, mrr_10

Common pitfalls

• The retrieval candidate pool is extremely large (26M), causing inherently low recall scores even when top-ranked answers are informative.
• Generation metrics like BLEU/ROUGE heavily penalize long, medically accurate answers due to strict n-gram overlap requirements, underestimating model capability.
• Split ratios vary by data source (90/10 for encyclopedias/knowledge bases vs. 99/1 for consultant records), making cross-source comparisons sensitive to pool size and test set composition.

Evidence (verbatim from paper)

We use Recall@k and MRR@10 as evaluation indicators. Recall@k measures the percentage of top k retrieved passages that contain the answer. MRR@10 calculates the average of the inverse of the ranks at which the first relevant document was retrieved.

Citation

@misc{li2023huatuo26m,
  title={Huatuo-26M, a Large-scale Chinese Medical QA Dataset},
  author={Jianquan Li et al. (2023)},
  year={2023},
  note={arXiv:2305.01526}
}

• arXiv: 2305.01526