Ram H1200 Eval

ram-h1200-eval

RAM-H1200: A Unified Evaluation and Dataset on Hand Radiographs for Rheumatoid Arthritis — Songxiao Yang et al. (2026) (arXiv:2605.05616, 2026)

What this evaluates

Evaluates medical imaging models on hand radiographs for rheumatoid arthritis. It probes anatomical structure modeling through bone segmentation, fine-grained lesion detection via bone erosion segmentation, and clinical reasoning through ordinal scoring of erosion and joint space narrowing severity.

Datasets

• RAM-H1200 — total 1200; splits: train (-1), val (-1), test (-1); repo https://github.com/YSongxiao/RAM-H1200

Metrics

• DSC (primary) — range: [0, 1]
  • 2 * |A ∩ B| / (|A| + |B|), measuring overlap between prediction and ground truth masks.
• QWK (primary) — range: [-1, 1]
  • Cohen's quadratic weighted kappa for ordinal agreement, penalizing larger deviations between predicted and true severity scores more heavily.
• ACC — range: [0, 1]
  • Proportion of correctly classified instances out of total instances.
• MAE — range: other
  • Average of absolute differences between predicted and true scores.
• REC — range: [0, 1]
  • True positives divided by the sum of true positives and false negatives.
• PREC — range: [0, 1]
  • True positives divided by the sum of true positives and false positives.

Input / output format

Input: Hand radiograph images (and joint region masks for scoring tasks).

Output: Segmentation masks (for bone and erosion) or ordinal severity scores (0-3 for BE and JSN).

Scoring recipe

intersection = np.sum(pred_mask & gold_mask)
union = np.sum(pred_mask) + np.sum(gold_mask)
dsc = 2 * intersection / (union + 1e-6)

from sklearn.metrics import cohen_kappa_score
qwk = cohen_kappa_score(gold_scores, pred_scores, weights='quadratic')

Common pitfalls

• Patient-level splits are strictly enforced to prevent data leakage; image-level random splits would invalidate results.
• Class imbalance is severe for BE scoring, making standard accuracy misleading without balanced accuracy or QWK.
• Bone overlap regions require specialized overlap-aware metrics (DSCO, NSDO) to properly evaluate boundary delineation.

Evidence (verbatim from paper)

All experiments were conducted using patient-level train/validation/test splits to prevent data leakage across subsets. ... For segmentation tasks, we report Dice similarity coefficient (DSC), normalized surface Dice (NSD), volumetric overlap error (VOE), mean surface distance (MSD), recall (REC), and precision (PREC). For SvdH score classification, we report quadratic weighted kappa (QWK), mean absolute error (MAE), balanced accuracy (BACC), accuracy (ACC), within-one accuracy (W1-ACC), positive/negative sensitivity (P/N-SEN), and positive/negative accuracy (P/N-ACC).

Citation

@misc{yang2026ramh1200,
  title={RAM-H1200: A Unified Evaluation and Dataset on Hand Radiographs for Rheumatoid Arthritis},
  author={Songxiao Yang et al. (2026)},
  year={2026},
  note={arXiv:2605.05616}
}

• arXiv: 2605.05616