Medical Image Classification Eval

medical-image-classification-eval

Can Large Language Models Challenge CNNs in Medical Image Analysis? — Ahmed et al. (2025) (arXiv:2505.23503, 2025)

What this evaluates

Evaluates the diagnostic accuracy and computational efficiency of CNNs versus multimodal LLMs on medical imaging tasks. It probes whether vision-language models can match traditional convolutional networks in classifying chest X-rays, MRIs, and CT scans, while also measuring prediction calibration and resource consumption.

Datasets

• Chest X-ray — total ?; splits: test (-1)
• Brain MRI — total ?; splits: test (-1)
• Chest CT — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Percentage of correctly classified samples out of the total number of samples.
• F1-Score — range: [0, 1]
  • Harmonic mean of precision and recall, providing a balanced measure of performance.
• Confidence Score — range: [0, 1]
  • Model's predicted probability or reliability score for each classification, analyzed using calibration curves.
• Execution Time — range: seconds
  • Wall-clock time required for model inference per sample or batch.
• Energy Consumption — range: watt-hours
  • Total electrical energy used during inference, calculated from execution time and power usage.

Input / output format

Input: Raw medical images (chest X-rays, brain MRIs, or chest CT scans) provided to either a CNN or a multimodal LLM (GPT-4o, Llama3.2-vision). In filtered experiments, images are pre-processed with an enhanced data filtering step to extract contextual details before LLM inference.

Output: Class prediction label, model confidence score, and system-level metrics (execution time in seconds, energy consumption in watt-hours, CO2 emissions).

Scoring recipe

def compute_metrics(predictions, ground_truth):
    correct = sum(1 for p, g in zip(predictions, ground_truth) if p == g)
    accuracy = correct / len(ground_truth)
    
    tp = sum(1 for p, g in zip(predictions, ground_truth) if p == g and g == 1)
    fp = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, ground_truth) if p == 0 and g == 1)
    
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    
    return {'accuracy': accuracy, 'f1': f1, 'precision': precision, 'recall': recall}

Common pitfalls

• LLMs exhibit severe miscalibration, reporting high confidence scores (e.g., >0.90) despite low accuracy (<0.65).
• Direct image classification by LLMs significantly underperforms CNNs unless augmented with domain-specific filtering or feature extraction layers.
• Computational efficiency metrics (time, energy, CO2) are highly hardware-dependent and not standardized across runs, making cross-study comparisons difficult.

Evidence (verbatim from paper)

Accuracy is measured by the percentage of correctly classified samples, providing an overall performance indicator. Precision assessed the proportion of true positives among predicted positives, minimizing false positives, while Recall measured the proportion of true positives among actual positives, emphasizing the model’s ability to identify relevant cases. The F1-Score, as the harmonic mean of precision and recall, offered a balanced measure of performance. Additionally, the Confidence Score analyzed prediction reliability using calibration curves. Finally, resource consumption, such as inference time and energy usage, was evaluated comprehensively to assess the models’ efficiency and practicality in real-world healthcare applications.

Citation

@misc{ahmed2025canlarge,
  title={Can Large Language Models Challenge CNNs in Medical Image Analysis?},
  author={Ahmed et al. (2025)},
  year={2025},
  note={arXiv:2505.23503}
}

• arXiv: 2505.23503