M Arc Eval

m-arc-eval

Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning — Kim et al. (2025) (arXiv:2502.04381, 2025)

What this evaluates

Probes inflexible reasoning and medical abstraction in LLMs by presenting adversarial, long-tail clinical scenarios designed to trigger the Einstellung effect. It evaluates whether models can apply deductive logic and uncertainty estimation rather than relying on rote pattern matching or memorization from pretraining data.

Datasets

• M-ARC — total 100; splits: test (100); repo https://github.com/dbernardo05/M-ARC

Metrics

• accuracy (primary) — range: percent
  • Percentage of correct answers out of 100 multiple-choice questions. Calculated as (number of correct predictions / total questions) * 100.
• Brier score — range: [0, 1]
  • Measures calibration by averaging the squared difference between predicted probability and actual outcome (0 or 1). Lower values indicate better calibration.
• sample consistency — range: [0, 1]
  • Inter-response agreement calculated by running the same question 15 times with slight variations (subject age varied by up to 10 days) and measuring the proportion of identical outputs.

Input / output format

Input: Multiple-choice clinical vignette questions modeled after USMLE format, including open-ended options to seek more data.

Output: Model must select one answer choice from the provided options.

Scoring recipe

# Accuracy
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
accuracy = correct / len(gold)

# Sample Consistency (uncertainty)
# Run model 15 times per question with age perturbation
consistency = 1 - (len(set(outputs_for_q)) / 15)

# Brier Score (calibration)
brier = sum((p_prob - g) ** 2 for p_prob, g in zip(pred_probs, gold)) / len(gold)

Common pitfalls

• The dataset uses adversarial long-tail scenarios specifically designed to trigger the Einstellung effect, so standard medical QA benchmarks may not reflect performance here.
• Uncertainty is measured via sample consistency (15 runs with age perturbation), not token-level probabilities or confidence elicitation, which differs from common LLM evaluation practices.
• Open-ended seek-more-data options are included in 53% of questions, requiring models to recognize insufficient information rather than force a clinical answer.

Evidence (verbatim from paper)

The accuracy of GPT-4o, o1, Medalpaca, Meditron-7b, Claude-Sonnet, Claude-Opus, Google Gemini, and Mistral models were evaluated. ... To assess model calibration, we utilized reliability plots and calculated the Brier score, following Lyu et al. ... inter-response agreement (consistency) is calculated as the uncertainty measure.

Citation

@misc{kim2025limitations,
  title={Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning},
  author={Kim et al. (2025)},
  year={2025},
  note={arXiv:2502.04381}
}

• arXiv: 2502.04381