Rjua QA Eval

rjua-qa-eval

RJUA-QA: A Comprehensive QA Dataset for Urology — Lyu et al. (2023) (arXiv:2312.09785, 2023)

What this evaluates

This benchmark evaluates large language models' ability to perform medical logical reasoning and urological disease diagnosis. It probes the model's capacity to handle complex, real-world clinical scenarios involving subjective patient queries and multi-disease comorbidity reasoning.

Datasets

• RJUA-QA — total 2132; splits: test (2132); repo https://github.com/alipay/RJU_Ant_QA

Metrics

• F1 score (diagnosis & advice) (primary) — range: [0, 1]
  • F1 = 2 × (P × R) / (P + R), where P = TP / (TP + FP) and R = TP / (TP + FN). The final accuracy is a weighted sum of the F1 scores for diagnosis and advice, using weights of 2/3 and 1/3 respectively.
• Rouge-L — range: [0, 1]
  • Computes precision, recall, and F1 based on the longest common subsequence (LCS) between the generated response and the reference text.

Input / output format

Input: Patient queries describing subjective symptoms and clinical scenarios.

Output: Diagnostic responses containing predicted diseases and examination advice.

Scoring recipe

def compute_metrics(predictions, gold):
    f1_diag = f1_score(predictions['disease'], gold['disease'])
    f1_adv = f1_score(predictions['advice'], gold['advice'])
    final_acc = (2/3) * f1_diag + (1/3) * f1_adv
    rouge_l = rouge_l_score(predictions['response'], gold['response'])
    return {'final_accuracy': final_acc, 'rouge_l': rouge_l}

Common pitfalls

• F1 is computed separately for diagnosis and advice, then combined with a fixed 2:1 weight ratio rather than a single unified F1.
• Rouge-L relies on longest common subsequence matching, which may penalize semantically correct but lexically varied medical advice.
• The protocol does not specify how multi-label comorbidity predictions are aggregated for F1 calculation (e.g., micro vs. macro averaging).

Evidence (verbatim from paper)

The F1 score is utilized to measure the accuracy for LLMs’ diagnosis and treatment. According to Table 1, F1 score is is formulated as: F1=2×(P×R)/(P+R), where P=TP/(TP+FP) denotes the precision and R=TP/(TP+FN) denotes the recall. A weighted sum of F1 score for diagnosis and advice is adopted to obtain the final accuracy, i.e., 2/3 for diagnosis and 1/3 for advice. To evaluate the overall quality of the LLMs’ responses, Rouge-L is exploited to calculate the longest common sub-sequence (LCS) between the generation and reference.

Citation

@misc{lyu2023rjuaqa,
  title={RJUA-QA: A Comprehensive QA Dataset for Urology},
  author={Lyu et al. (2023)},
  year={2023},
  note={arXiv:2312.09785}
}

• arXiv: 2312.09785