Chinese Tibetan Medicine QA Eval

chinese-tibetan-medicine-qa-eval

Traceable Cross-Source RAG for Chinese Tibetan Medicine Question Answering — Fengxian Chen et al. (2026) (arXiv:2602.05195, 2026)

What this evaluates

This benchmark evaluates a traceable cross-source retrieval-augmented generation framework for Chinese Tibetan medicine QA. It probes the model's ability to route queries across heterogeneous knowledge bases, fuse cross-source evidence, and generate faithful answers with correct citations.

Datasets

• Chinese Tibetan-medicine QA dataset — total 500; splits: full (500)

Metrics

• CrossEv@5 (primary) — range: [0, 1]
  • Checks whether the top-5 retrieved evidence chunks cover all required knowledge bases for cross-KB queries. Computed as 1 if coverage is complete, 0 otherwise.
• Faithfulness — range: [0, 1]
  • Reference-free RAGAS-style metric measuring whether generated claims are fully supported by the retrieved context.
• Citation Correctness (CitCorr) — range: [0, 1]
  • ALCE-inspired check verifying whether each cited evidence chunk actually supports the corresponding generated statement.
• EvRecall@5 — range: [0, 1]
  • Fraction of gold evidence chunks retrieved within the top-5 ranked results.
• EvNDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5 for evidence chunk ranking.
• PrimaryAcc — range: [0, 1]
  • Accuracy of predicting the primary source knowledge base for a query.
• Top2Hit — range: [0, 1]
  • Fraction of queries where the primary source KB appears in the top-2 ranked KBs.
• EDR — range: [0, 1]
  • Encyclopedia Dominance Rate; fraction of encyclopedia chunks in the final evidence list.
• AuthCov — range: [0, 1]
  • Measures whether evidence from authoritative KBs appears in the top-5 results for relevant queries.
• Context Precision — range: [0, 1]
  • RAGAS-style metric measuring the precision of retrieved context relevant to the query.
• Context Recall — range: [0, 1]
  • RAGAS-style metric measuring the recall of retrieved context relative to the gold answer.
• Answer Relevance — range: [0, 1]
  • RAGAS-style metric measuring how relevant the generated answer is to the query.

Input / output format

Input: Query, gold answer, gold evidence at chunk level (KB labels, document metadata, stable chunk ids). For cross-KB synthesis queries, the required KB set and primary source KB are specified.

Output: Generated answer, cited evidence chunks, KB routing scores/rankings, and evidence ranking list.

Scoring recipe

# CrossEv@5 (cross-KB subset only)
coverage = len(set(e.kb for e in top_5_evidence) & required_kbs) == len(required_kbs)
cross_ev = coverage

# EvRecall@5 & EvNDCG@5
gold_ids = set(gold_chunk_ids)
pred_ids = set(pred_chunk_ids[:5])
ev_recall = len(gold_ids & pred_ids) / len(gold_ids)
ev_ndcg = dcg_at_k(pred_ids, k=5) / idcg_at_k(gold_ids, k=5)

# Faithfulness & CitCorr (LLM-judged)
faithfulness = judge_model.check_claims_supported(answer, context)
cit_corr = judge_model.check_citations_support_statements(answer, citations)

Common pitfalls

• No train/dev/test splits exist; all 500 queries are evaluated together, so metrics reflect a single fixed configuration rather than split-averaged performance.
• Evidence-level metrics (EvRecall@5, EvNDCG@5, CrossEv@5) are computed exclusively on the cross-KB subset, not the full dataset.
• All automatic metrics are computed by a single judge model (GLM-4.7) to avoid evaluator heterogeneity, which may introduce systematic judge bias.
• A fixed cutoff of K=5 is used for all @K metrics, regardless of query type or retrieval depth requirements.

Evidence (verbatim from paper)

We evaluate at three levels with a consistent cutoff K=5. We report EvRecall@5 and EvNDCG@5 based on gold chunk ids. For cross-KB queries, we report CrossEv@5, which checks whether the top-5 evidence covers all required KBs. Unless otherwise stated, evidence-level metrics are computed over the cross-KB subset. We report reference-free RAG metrics (faithfulness, context precision/recall, and answer relevance) in the spirit of RAGAS. We also report citation-based checks inspired by ALCE to assess whether cited evidence supports the generated statements.

Citation

@misc{chen2026traceable,
  title={Traceable Cross-Source RAG for Chinese Tibetan Medicine Question Answering},
  author={Fengxian Chen et al. (2026)},
  year={2026},
  note={arXiv:2602.05195}
}

• arXiv: 2602.05195