Gemex Eval

gemex-eval

GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis — Liu et al. (2024) (arXiv:2411.16778, 2024)

What this evaluates

Evaluates large vision-language models on chest X-ray diagnosis by testing their ability to answer medical questions, provide textual reasoning, and ground answers to specific visual regions in radiographs.

Datasets

• GEMeX — total ?; splits: train (-1), test (-1)

Metrics

• AR-score (primary) — range: percent
  • Relative GPTScore calculated by GPT-4o. The reference (ground truth answer + reason) and candidate (model output) are scored on a 1-10 scale for accuracy, relevance, and helpfulness, then normalized relatively to a percentage.
• A-score — range: percent
  • Exact match accuracy for closed-ended, single-choice, and multi-choice questions. Compares the model's extracted answer against the ground truth option(s).
• V-score — range: percent
  • Mean Intersection over Union (mIoU) for visual grounding. Predicted bounding boxes are matched to ground truth boxes using the Hungarian algorithm before computing IoU.

Input / output format

Input: Chest X-ray image paired with a medical question. Questions are categorized as open-ended, closed-ended, single-choice, or multi-choice.

Output: Textual answer, detailed reasoning, and corresponding visual location (bounding box coordinates).

Scoring recipe

def compute_metrics(predictions, gold):
    # AR-score (GPTScore)
    ar_score = gpt4o_score(reference=f"{gold['answer']} {gold['reason']}",
                            candidate=predictions['text'],
                            question=gold['question'],
                            report=gold['report'])
    # A-score (Accuracy)
    a_score = 100.0 if predictions['answer'] == gold['answer'] else 0.0
    # V-score (mIoU with Hungarian matching)
    pred_boxes = parse_boxes(predictions['location'])
    gt_boxes = gold['location']
    matches = hungarian_match(pred_boxes, gt_boxes)
    ious = [compute_iou(p, g) for p, g in matches]
    v_score = 100.0 * (sum(ious) / len(gt_boxes) if gt_boxes else 0)
    return {'AR-score': ar_score, 'A-score': a_score, 'V-score': v_score}

Common pitfalls

• Models often fail to follow strict output formats, leading to low AR-scores despite having correct medical knowledge.
• Standard NLG metrics (BLEU, ROUGE) poorly correlate with actual reasoning quality, especially when outputs are short or omit reasoning.
• Models tend to use shortcut reasoning from pre-training rather than actual visual grounding, particularly on choice-based questions.

Evidence (verbatim from paper)

In GEMeX, each question has a corresponding answer, textual reason, and visual location. Ideally, we aim to evaluate all these three aspects with designed metrics as follows: • Answer-Reason Score (AR-score): In reality, most LVLMs struggle to generate accurate outputs in terms of format. This doesn’t necessarily mean these models lack the knowledge to answer the questions but rather simply lack the ability to follow instructions properly. To ensure a fair comparison, we introduce the Answer-Reason score (AR-score) as an evaluation metric for the textual output, where the answer and reason parts from each test sample are merged as a reference (ground truth), and the evaluated LVLM’s output serve as a candidate. We use GPTScore to calculate the AR-Score from a semantic perspective.

Citation

@misc{liu2024gemex,
  title={GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2411.16778}
}

• arXiv: 2411.16778