qhjqhj00 redis-qa-eval
Assessing and Enhancing Large Language Models in Rare Disease Question-answering — Wang et al. (2024) (arXiv:2408.08422, 2024)
What this evaluates
Evaluates large language models' ability to answer questions about rare diseases, including diagnosis, symptoms, causes, and related properties. It probes the models' medical knowledge retrieval and reasoning capabilities in a specialized, low-resource domain.
Datasets
- ReDis-QA — total 1360; splits: test (-1); repo https://github.com/guanchuwang/redis-bench
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions out of the total number of instances. Calculated as the number of correct predictions divided by the total number of questions, multiplied by 100.
Input / output format
Input: Natural language question about a rare disease (e.g., symptoms, causes, diagnosis, related disorders), provided via instruction-tuned prompts.
Output: Natural language answer to the question.
Scoring recipe
correct_count = 0
for pred, gold in zip(predictions, gold_answers):
if normalize(pred) == normalize(gold):
correct_count += 1
accuracy = (correct_count / len(gold_answers)) * 100
Common pitfalls
- Model performance varies significantly across different disease property categories (e.g., symptoms vs. diagnosis), so reporting only aggregate accuracy may mask domain-specific weaknesses.
- The evaluation relies on specific instruction-tuned prompts detailed in Appendix 1, meaning results are highly sensitive to prompt formatting and system instructions not fully described in the main text.
Evidence (verbatim from paper)
The evaluation metric is the accuracy on the ReDis-QA dataset. ... Figure 2 (b) illustrates the accuracy of LLMs in percentage, with accuracy for each subset of properties displayed separately.
Citation
@misc{wang2024redisqa,
title={Assessing and Enhancing Large Language Models in Rare Disease Question-answering},
author={Wang et al. (2024)},
year={2024},
note={arXiv:2408.08422}
}
- arXiv: 2408.08422