RAG Medical Eval

rag-medical-eval

Pursuing Best Industrial Practices for Retrieval-Augmented Generation in the Medical Domain — Zhu (2026) (arXiv:2602.03368, 2026)

What this evaluates

Evaluates the effectiveness and efficiency of Retrieval-Augmented Generation (RAG) systems across medical and general knowledge domains. It probes how different RAG pipeline components (chunking, indexing, query classification, augmentation, and prompting) impact answer accuracy and response latency on question-answering and information extraction tasks.

Datasets

• MMLU — total ?; splits: test (-1); HF cais/mmlu
• PubMedQA — total ?; splits: test (-1); HF pubmedqa
• PromptNER — total ?; splits: test (-1)
• Query Classification Dataset — total 27900; splits: train (24000), dev (2000), test (1900)

Metrics

• accuracy (acc) (primary) — range: [0, 1]
  • Proportion of correctly predicted final answers out of total test samples for MMLU and PubMedQA.
• instance-level strict micro-F1 — range: [0, 1]
  • Model predicts an entity correctly if and only if it correctly predicts all its components. Computed as micro-averaged F1 across all instances for PromptNER.
• average latency — range: seconds
  • Average time in seconds required to complete the full response generation for a single test sample.

Input / output format

Input: Query (question/instruction) optionally augmented with retrieved top-k=8 document segments, formatted according to the prompting strategy (e.g., COT-Refine, direct answering, or query rewriting).

Output: Final answer text for QA tasks; JSON-formatted entity list for PromptNER (parsed from response text, defaults to null list if unparseable); binary label (0 or 1) for query classification.

Scoring recipe

def compute_metrics(predictions, golds, latencies, task):
    if task in ['MMLU', 'PubMedQA']:
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    elif task == 'PromptNER':
        parsed_preds = [parse_json(p) if is_valid_json(p) else [] for p in predictions]
        return micro_f1_strict(parsed_preds, golds)
    elif task == 'latency':
        return sum(latencies) / len(latencies)

Common pitfalls

• PromptNER predictions must be strictly parsed into JSON; unparseable outputs are treated as empty lists, which can artificially lower F1 if parsers are not consistent.
• Query classification uses an arbitrary log-likelihood threshold (l1 - l0 > 0) to label 'need RAG', which may not align with optimal retrieval boundaries or human judgment.
• Average latency measures the full RAG pipeline (retrieval + generation), making direct comparisons with the 'No RAG' baseline sensitive to retrieval overhead rather than just generation speed.

Evidence (verbatim from paper)

For the MMLU and PubMedQA tasks, we will directly consider the correctness of the final answers. Thus, we report accuracy (denoted as acc). For the PromptNER task, the output response text will first be parsed and transformed to a json instance. If the response can not be parsed to json, then we consider the prediction as a null list. We adopt the instance-level strict micro-F1 following Zhu et al. (2023f), that is, the model predicts an entity correctly if and only if it correctly predicts all its components. Other than the performance matrices on the evaluation datasets, we also measure the efficiency of the RAG systems by the average latency (in seconds (s)) for completing the response for a test sample.

Citation

@misc{zhu2026pursuing,
  title={Pursuing Best Industrial Practices for Retrieval-Augmented Generation in the Medical Domain},
  author={Zhu (2026)},
  year={2026},
  note={arXiv:2602.03368}
}

• arXiv: 2602.03368