# Score

> Audits medical LLM benchmarks across five lifecycle phases using 46 medically tailored criteria to assess clinical relevance, data integrity, safety-critical capabilities, validity, and governance.

- Skill: `qhjqhj00/score` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/score`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/score/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML, Research & Search, Literature Review, Model Training & Fine-tuning
- Tags: Benchmark Audit, Clinical Relevance, Data Integrity, Governance, Llm Evaluation, Medical Benchmark
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-08-22
- Page: https://skillmd.com/skills/qhjqhj00/score

---


# score

> Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models — Ma et al. (2025) (arXiv:2508.04325, 2025)

## What this evaluates

This framework audits medical LLM benchmarks across five lifecycle phases using 46 medically tailored criteria to assess clinical relevance, data integrity, safety-critical capabilities, validity, and governance.

## Datasets

- (no dataset; pure metric skill)

## Metrics

- `score` **(primary)** — range: percent
  - Count of satisfied criteria out of 46 total across five phases (Design, Dataset Construction, Technical Implementation, Benchmark Validity, Documentation/Governance), reported as a percentage.

## Input / output format

**Input**: Documentation and design details of a medical LLM benchmark to be audited.

**Output**: Binary or categorical assessment for each of the 46 MedCheck criteria.

## Scoring recipe

```python
def compute_score(benchmark):
    satisfied = sum(1 for c in criteria if benchmark.meets(c))
    return (satisfied / 46) * 100
```

## Common pitfalls

- Scoring relies on qualitative expert judgment rather than automated computation, risking subjectivity.
- The score evaluates benchmark design quality, not model inference performance, so it cannot replace standard accuracy metrics.

## Evidence (verbatim from paper)

> This phase is the most underdeveloped in our analysis (average score: 52.4%), revealing a profound gap between current practice and the requirements for reliable medical AI.

## Citation

```bibtex
@misc{ma2025beyond,
  title={Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models},
  author={Ma et al. (2025)},
  year={2025},
  note={arXiv:2508.04325}
}
```

- arXiv: 2508.04325