Multimed Eval

multimed-eval

MultiMed: Massively Multimodal and Multitask Medical Understanding — Mo et al. (2024) (arXiv:2408.12682, 2024)

What this evaluates

Evaluates multimodal medical understanding across 11 diverse tasks including disease classification, imaging analysis, genomics, proteomics, and medical VQA. Probes cross-modal integration, generalization to out-of-distribution organs and cell types, and robustness to few-shot and zero-shot scenarios.

Datasets

• MultiMed — total 2560000; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Computed as the proportion of correct predictions over the total number of instances, multiplied by 100. Results are averaged over three independent training runs with different random seeds.
• Pearson correlation — range: [-1, 1]
  • Measures the linear correlation coefficient between predicted and actual continuous gene expression values. It evaluates the precision of quantitative outputs when predictions vary in scale.
• TM score — range: [0, 1]
  • Template Modeling score that compares predicted 3D protein structures against ground truth to assess the similarity in the spatial arrangement of the protein backbone. Values closer to 1 indicate higher structural quality.

Input / output format

Input: Multimodal medical data instances tailored to specific tasks, including imaging (X-ray, MRI, CT), genomic sequences, scRNA-seq data, EEG time-series, protein sequences, or clinical text, paired with task-specific labels.

Output: Task-specific predictions: categorical class labels for classification tasks, continuous scalar values for gene expression prediction, and 3D coordinate arrays for protein structure prediction.

Scoring recipe

def compute_metric(predictions, gold, task_type):
    if task_type == 'classification':
        return (predictions == gold).mean() * 100
    elif task_type == 'expression':
        return pearsonr(predictions, gold).statistic
    elif task_type == 'structure':
        return tm_score(predictions, gold)
    return None
# Average the per-task metric over 3 random seeds for final reporting.

Common pitfalls

• Averaging accuracy across all 11 tasks versus reporting per-task percentages; the paper reports per-task values in Table 1 but references an 'average accuracy score' in the text, requiring careful alignment when reproducing.
• TM score interpretation differs from standard loss metrics: higher values (closer to 1) indicate better performance, not worse.
• Results must be averaged over three independent runs with different random seeds; single-run reports will not match the paper's protocol.

Evidence (verbatim from paper)

To assess performance across these datasets, we employ the average accuracy score on the test set, computed over three runs with different seeds. Accuracy is particularly suitable for tasks where outcomes are categorical and labels are balanced. For gene expression prediction, we adopt the Pearson correlation score. This measure evaluates the linear correlation between the predicted and actual gene expressions, offering insight into the precision of the model’s quantitative outputs. It is especially relevant in this context as gene expression data is continuous and predictions can vary in scale. For protein structure prediction, we use the TM score (Template Modeling score) to evaluate the quality of the predicted 3D structure.

Citation

@misc{mo2024multimed,
  title={MultiMed: Massively Multimodal and Multitask Medical Understanding},
  author={Mo et al. (2024)},
  year={2024},
  note={arXiv:2408.12682}
}

• arXiv: 2408.12682