Openmedia Eval

openmedia-eval

OpenMedIA: Open-Source Medical Image Analysis Toolbox and Benchmark under Heterogeneous AI Computing Platforms — Zhuang et al. (2022) (arXiv:2208.05616, 2022)

What this evaluates

This benchmark evaluates the performance and cross-framework compatibility of deep learning algorithms for medical image analysis across classification, segmentation, localization, and detection tasks. It specifically probes how model accuracy and inference efficiency vary when implementations are ported between PyTorch and MindSpore on heterogeneous hardware (NVIDIA GPUs vs. Huawei Ascend NPUs).

Datasets

• OpenMedIA Benchmark Suite — total ?; splits: test (-1)

Metrics

• Accuracy (Acc) (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances. Calculated as correct predictions divided by total predictions.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall. Calculated as 2 * (precision * recall) / (precision + recall).
• Dice Score — range: [0, 1]
  • Overlap coefficient between predicted and ground truth segmentation masks. Calculated as 2 * |A ∩ B| / (|A| + |B|).
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve, measuring the model's ability to distinguish between classes across all classification thresholds.
• mAP — range: [0, 1]
  • Mean Average Precision across all object classes or detection tasks. Calculated as the average of the Area Under the Precision-Recall curve for each class.

Input / output format

Input: Medical images in 2D or 3D formats (CT scans, ultrasound, endoscopy, histology) provided as input tensors to the model.

Output: Task-specific predictions: class labels for classification, binary/semantic masks for segmentation, bounding boxes or rotated ellipses for detection, and localization scores for weakly supervised tasks.

Scoring recipe

def compute_metrics(predictions, ground_truth, task_type):
    if task_type == 'classification':
        return {'Acc': (predictions == ground_truth).mean(),
                'F1': f1_score(ground_truth, predictions)}
    elif task_type == 'segmentation':
        intersection = np.sum(predictions * ground_truth)
        union = np.sum(predictions) + np.sum(ground_truth)
        return {'Dice': 2 * intersection / (union + 1e-6)}
    elif task_type == 'localization':
        return {'AUC': roc_auc_score(ground_truth, predictions)}
    elif task_type == 'detection':
        return {'mAP': compute_map(predictions, ground_truth)}
    return None

Common pitfalls

• Comparing inference times across different hardware architectures (NVIDIA Tesla V100 vs. Huawei Ascend 910) without accounting for framework-specific optimizations or batch size differences.
• Using Accuracy as the sole metric for segmentation or detection tasks, which ignores spatial overlap or localization precision captured by Dice or mAP.
• Assuming framework implementations (PyTorch vs. MindSpore) are directly comparable without verifying identical data augmentation, preprocessing, and hyperparameter settings.

Evidence (verbatim from paper)

For 2D classification task, accuracy(Acc) and F1 are used to evaluate the classification accuracy. For 2D/3D segmentation tasks, Dice score is used to measure the segmentation accuracy. For weakly supervised lesion localisation and segmentation tasks, AUC and Dice score are used for quantitative comparisons. For 2D detection tasks, mAP , AUC and Dice score are used for evaluation.

Citation

@misc{zhuang2022openmedia,
  title={OpenMedIA: Open-Source Medical Image Analysis Toolbox and Benchmark under Heterogeneous AI Computing Platforms},
  author={Zhuang et al. (2022)},
  year={2022},
  note={arXiv:2208.05616}
}

• arXiv: 2208.05616