Mimic Vqa Eval

mimic-vqa-eval

Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning — Hu et al. (2023) (arXiv:2302.09636, 2023)

What this evaluates

Evaluates a model's ability to answer clinical questions about chest X-rays, focusing on disease presence, type, location, and severity. It probes multi-modal reasoning by requiring the model to correlate image regions with structured medical knowledge and spatial/semantic relationships.

Datasets

• Mimic-VQA — total 297723; splits: train (-1), val (-1), test (-1)

Metrics

• AUC-micro (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. AUC-micro computes the final AUC by aggregating the contributions of each class across all samples before computing the curve.
• AUC-macro — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. AUC-macro treats all classes equally and computes the average AUC across the 169 answer categories.

Input / output format

Input: A chest X-ray image and a natural language question (e.g., about abnormality, presence, location, type, or level).

Output: A set of predicted answer scores/probabilities for 169 possible answer classes, with top predictions selected if score > 0.04.

Scoring recipe

def compute_auc(y_true, y_pred, average='micro'):
    if average == 'micro':
        fpr, tpr, _ = roc_curve(y_true.ravel(), y_pred.ravel())
        return auc(fpr, tpr)
    else:
        fprs, tprs = [], []
        for i in range(y_true.shape[1]):
            fpr, tpr, _ = roc_curve(y_true[:, i], y_pred[:, i])
            fprs.append(fpr)
            tprs.append(tpr)
        mean_fpr = np.linspace(0, 1, 100)
        mean_tpr = np.interp(mean_fpr, fprs[0], tprs[0])
        for fpr, tpr in zip(fprs[1:], tprs[1:]):
            mean_tpr = np.maximum(mean_tpr, np.interp(mean_fpr, fpr, tpr))
        return auc(mean_fpr, mean_tpr)

Common pitfalls

• Sequential train/val/test split (8:1:1) may cause data leakage or distribution shift if images from the same study appear across splits.
• AUC-micro vs AUC-macro handle class imbalance differently; micro favors majority classes while macro treats all 169 answer classes equally.
• The dataset filters rare answers to 169 classes, which may not reflect real-world long-tail disease distributions.

Evidence (verbatim from paper)

We used the AUC as the evaluation metric. AUC-micro computes the final AUC by aggregating the contributions of each class. AUC-macro treats all classes equally and computes the average AUC.

Citation

@misc{hu2023interpretable,
  title={Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning},
  author={Hu et al. (2023)},
  year={2023},
  note={arXiv:2302.09636}
}

• arXiv: 2302.09636