Wingpt 3.0 Benchmark Eval

wingpt-3.0-benchmark-eval

WiNGPT-3.0 Technical Report — Zhuang et al. (2025) (arXiv:2505.17387, 2025)

What this evaluates

Evaluates large language models on comprehensive medical reasoning, clinical calculation, and general cognitive capabilities. It probes domain-specific knowledge application, diagnostic reasoning, and complex problem-solving in real-world clinical and academic settings.

Datasets

• MedCalc — total 2811; splits: test (2811)
• MedReMCQ — total 700; splits: test (700)
• CMMLU — total 11582; splits: test (11582)
• MATH-500 — total 500; splits: test (500)
• MedQA-USMLE — total 1273; splits: test (1273)
• MedMCQA — total 4183; splits: test (4183)
• PubMedQA — total 1000; splits: test (1000)

Metrics

• accuracy (primary) — range: [0, 1]
  • Average of 1 if the model's prediction matches the gold answer (or falls within the specified upper/lower bounds for MedCalc), else 0.
• Micro-F1 score — range: [0, 1]
  • Micro-averaged F1 score computed across all classes/options for the MedReMCQ dataset.

Input / output format

Input: Multiple-choice questions or open-ended clinical/mathematical prompts requiring reasoning or calculation.

Output: Selected option letter/answer for MCQs, or computed numerical/textual answer for calculation/reasoning tasks.

Scoring recipe

def score_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold: # or within bounds for MedCalc
            correct += 1
    return correct / len(golds)

def score_micro_f1(predictions, golds):
    tp = fp = fn = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold: tp += 1
        elif pred != gold: fp += 1
        if gold != pred: fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• MedCalc uses a tolerance-based accuracy metric where answers must fall within upper/lower bounds of the standard answer, not exact string match.
• Micro-F1 is used for MedReMCQ instead of standard accuracy, requiring class-level aggregation across all options.
• Benchmarks are evaluated on their test splits without explicit train/val splits mentioned in the report.

Evidence (verbatim from paper)

MedCalc uses accuracy as the metric which is based on whether the model’s computed answer falls within the upper and lower bounds of the standard answer. MedReMCQ is a dataset we developed in-house, tailored to medical reasoning scenarios. It is presented in a multiple-choice format and assessed using the Micro-F1 score. CMMLU, MATH-500, MedQA-USMLE, MedMCQA and PubMedQA use accuracy as the evaluation metric.

Citation

@misc{zhuang2025wingpt3,
  title={WiNGPT-3.0 Technical Report},
  author={Zhuang et al. (2025)},
  year={2025},
  note={arXiv:2505.17387}
}

• arXiv: 2505.17387