Schain Medical Reasoning Eval

schain-medical-reasoning-eval

S-Chain: Structured Visual Chain-of-Thought For Medicine — Khai Le-Duc et al. (2025) (arXiv:2510.22728, 2025)

What this evaluates

Evaluates medical vision-language models on disease classification and structured visual reasoning. It probes the model's ability to localize lesions, generate clinically faithful chain-of-thought rationales, and produce accurate diagnostic classifications grounded in visual evidence.

Datasets

• S-Chain — total 12000; splits: train (10783), test (1542)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Percentage of correctly predicted disease classes in the final diagnostic task (Q4).
• F1 — range: [0, 1]
  • F1 score for disease classification (Q4), calculated to capture class balance across diagnostic categories.
• mIoU — range: [0, 1]
  • Mean Intersection over Union for bounding box localization (Q1), measuring overlap between predicted and ground-truth lesion regions.
• BLEU / METEOR / BERTScore — range: [0, 1]
  • Standard text similarity metrics used to evaluate the semantic faithfulness and clinical plausibility of generated chain-of-thought descriptions (Q2).
• Lesion Grading Accuracy — range: [0, 1]
  • Accuracy of predicted lesion severity grades against expert-annotated scores (Q3).

Input / output format

Input: Medical image paired with a structured question (Q1-Q4). For RAG settings, top-k retrieved medical passages are concatenated. For controlled experiments, ground-truth intermediate steps (bounding boxes, CoT text, grading) may be provided as input.

Output: For Q1: bounding box coordinates. For Q2: chain-of-thought text description. For Q3: lesion grade score. For Q4: final diagnostic classification label.

Scoring recipe

def score(predictions, gold):
    # Q4 Classification
    acc = sum(1 for p, g in zip(predictions['q4'], gold['q4']) if p == g) / len(gold['q4'])
    f1 = f1_score(gold['q4'], predictions['q4'], average='macro')
    # Q1 Localization
    miou = mean_iou(predictions['q1_boxes'], gold['q1_boxes'])
    # Q2 Text
    bleu = bleu_score(predictions['q2_text'], gold['q2_text'])
    meteor = meteor_score(predictions['q2_text'], gold['q2_text'])
    bertscore = bertscore_f1(predictions['q2_text'], gold['q2_text'])
    # Q3 Grading
    grade_acc = sum(1 for p, g in zip(predictions['q3'], gold['q3']) if p == g) / len(gold['q3'])
    return {'Accuracy': acc, 'F1': f1, 'mIoU': miou, 'BLEU': bleu, 'METEOR': meteor, 'BERTScore': bertscore, 'Grading_Acc': grade_acc}

Common pitfalls

• High final diagnostic accuracy may mask unfaithful or hallucinated reasoning traces that ignore visual evidence.
• Synthetic CoT training often inherits teacher model hallucinations, leading to missing or misaligned bounding boxes.
• Textual supervision of bounding box coordinates yields weaker reasoning alignment compared to visual prompting (highlighted ROIs).

Evidence (verbatim from paper)

Task and Metrics: We evaluate models primarily on disease classification (Q4), reporting both Accuracy and F1 to capture overall correctness and class balance. Intermediate steps are also assessed: bounding box localization (Q1) with mIoU, lesion grading (Q3) with Accuracy against expert scores, and CoT descriptions (Q2) with BLEU, METEOR, BERTScore for semantic similarity for faithfulness and clinical plausibility.

Citation

@misc{leduc2025schain,
  title={S-Chain: Structured Visual Chain-of-Thought For Medicine},
  author={Khai Le-Duc et al. (2025)},
  year={2025},
  note={arXiv:2510.22728}
}

• arXiv: 2510.22728