Covid Blues Eval

covid-blues-eval

COVID-BLUeS -- A Prospective Study on the Value of AI in Lung Ultrasound Analysis — Wiedemann et al. (2025) (arXiv:2509.10556, 2025)

What this evaluates

This benchmark evaluates AI models for detecting COVID-19 infection and assessing lung severity using lung ultrasound videos, clinical variables, and blood count data. It measures how well zero-shot and fine-tuned models generalize to real-world, heterogeneous clinical data compared to human annotators and tabular baselines.

Datasets

• COVID-BLUeS — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/NinaWie/COVID-BLUES

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified patients (TP+TN)/Total. Reported alongside sensitivity, specificity, and F1-score.
• sensitivity — range: [0, 1]
  • True positive rate: TP/(TP+FN).
• specificity — range: [0, 1]
  • True negative rate: TN/(TN+FN).
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall: 2*(Precision*Recall)/(Precision+Recall).
• MAE — range: other
  • Mean Absolute Error between predicted and actual severity scores.

Input / output format

Input: Lung ultrasound video frames (processed frame-by-frame or via segmentation/localization), optionally concatenated with 55 clinical variables and 11 complete blood count (CBC) features.

Output: Binary patient-level classification (COVID-19 positive/negative) or ordinal severity score.

Scoring recipe

# For binary COVID-19 detection (patient-level)
patient_preds = aggregate_frame_predictions(frame_outputs, aggregation='model_output')
accuracy = sum(pred == gold for pred, gold in zip(patient_preds, gold_labels)) / len(gold_labels)
sensitivity = TP / (TP + FN)
specificity = TN / (TN + FN)
f1 = 2 * precision * recall / (precision + recall)
# For severity scoring
mae = mean(abs(pred_score - true_score))
rmse = sqrt(mean((pred_score - true_score)**2))

Common pitfalls

• Frame-level processing ignores video-level temporal structure, leading to suboptimal performance compared to segmentation-based aggregation.
• Prior studies often use biased datasets with manually selected clear-pathology frames, inflating reported zero-shot accuracy.
• Clinical variables alone (e.g., CBC data) can outperform image-only AI models, making image-only benchmarks clinically misleading.

Evidence (verbatim from paper)

The best performance was achieved using the ICLUS segmentation model combined with logistic regression, yielding an accuracy of 79%. However, even this model falls significantly short of the higher accuracies reported in related work. These findings highlight the substantial challenges in accurately classifying the COVID-BLUeS dataset, regardless of the method—whether through human assessment, zero-shot application of AI models, CNN training, or pattern detection techniques.

Citation

@misc{wiedemann2025covidblues,
  title={COVID-BLUeS -- A Prospective Study on the Value of AI in Lung Ultrasound Analysis},
  author={Wiedemann et al. (2025)},
  year={2025},
  note={arXiv:2509.10556}
}

• arXiv: 2509.10556