Amega Clinical Reasoning Eval

amega-clinical-reasoning-eval

Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning — Nissen et al. (2025) (arXiv:2502.08954, 2025)

What this evaluates

Evaluates the clinical reasoning capabilities and on-device runtime efficiency of various LLMs using the AMEGA benchmark. It measures response accuracy via an LLM-as-a-judge scoring system and tracks inference throughput and thermal throttling effects across different mobile hardware configurations.

Datasets

• AMEGA — total ?; splits: test (-1); repo https://github.com/StanfordBDHG/HealthBench

Metrics

• AMEGA score (primary) — range: other
  • Mean score assigned by GPT-4o to model-generated responses on clinical cases and questions, averaged across all compatible device runs.
• output tokens per second — range: tokens/sec
  • Inference throughput calculated as the number of generated output tokens divided by total inference time, reported as minimum and maximum values across tested devices.

Input / output format

Input: Clinical reasoning cases and questions from the AMEGA dataset, processed by on-device LLMs for text generation.

Output: Generated text responses from the LLMs, which are subsequently fed into GPT-4o for automated evaluation.

Scoring recipe

responses = run_models_on_devices(amega_dataset)
judge_scores = []
for resp in responses:
    score = gpt4o_judge(resp)
    judge_scores.append(score)
omega_score = mean(judge_scores)
throughput = count_tokens(resp) / inference_time

Common pitfalls

• Memory constraints on older iOS devices often prevent larger models from running, leading to missing data or forced exclusions (e.g., MedFound 8B).
• Thermal throttling significantly degrades throughput (up to 55% drop in 'serious' state), which must be accounted for when comparing runtime performance.
• iOS enforces a strict 2.25 GB memory limit that terminates applications exceeding it, regardless of theoretical model footprint.

Evidence (verbatim from paper)

Table 3 presents the mean AMEGA scores for each model, categorized by type and size. The mean score was computed across all model runs on compatible devices.

Citation

@misc{nissen2025medicineontheedge,
  title={Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning},
  author={Nissen et al. (2025)},
  year={2025},
  note={arXiv:2502.08954}
}

• arXiv: 2502.08954