Ecovnet Covidx Eval

ecovnet-covidx-eval

ECOVNet: An Ensemble of Deep Convolutional Neural Networks Based on EfficientNet to Detect COVID-19 From Chest X-rays — Chowdhury et al. (2020) (arXiv:2009.11850, 2020)

What this evaluates

Evaluates the ability of deep convolutional neural networks to classify chest X-ray images into three categories: COVID-19, normal, and pneumonia. It probes robustness under class imbalance and tests the effectiveness of ensemble learning strategies (hard vs. soft voting) combined with data augmentation.

Datasets

• COVIDx — total ?; splits: train (12525), validation (1389), test_balanced (300), test_imbalanced (1579)

Metrics

• accuracy (primary) — range: [0, 1]
  • Calculated as (TP + TN) / Total Samples. Represents the proportion of correctly classified instances out of all instances.
• precision — range: [0, 1]
  • Calculated as TP / (TP + FP). Measures the proportion of positive predictions that are actually correct.
• recall — range: [0, 1]
  • Calculated as TP / (TP + FN). Measures the proportion of actual positives that are correctly identified.
• F1 score — range: [0, 1]
  • Calculated as 2 * (Precision * Recall) / (Precision + Recall). The harmonic mean of precision and recall, preferred for imbalanced datasets.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve, which plots True Positive Rate (Recall) against False Positive Rate (FP / (FP + TN)) across different thresholds.

Input / output format

Input: Single chest X-ray image, optionally augmented, resized to model-specific resolution (224x224 to 456x456 depending on EfficientNet variant).

Output: Class label prediction (COVID-19, Normal, or Pneumonia) or class-wise probability scores for ensemble aggregation.

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob=None):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == p == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    acc = (tp + tn) / (tp + tn + fp + fn)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    auc = roc_auc_score(y_true, y_prob) if y_prob is not None else None
    return acc, prec, rec, f1, auc

Common pitfalls

• The natural test set is highly imbalanced; the paper explicitly constructs a separate 'balanced' test set (100 per class), so reporting results without specifying which test split was used is misleading.
• Soft ensemble averages softmax probabilities while hard ensemble uses majority voting; confusing the two reverses their performance ranking on certain splits.
• Confidence intervals are computed per test set; the balanced test set contains only 300 samples, yielding significantly wider CIs than the imbalanced set, which can be misinterpreted as lower model stability.

Evidence (verbatim from paper)

In order to evaluate the performance of the proposed method, we considered the following evaluation metrics: accuracy, precision, recall, F1 score, confidence interval (CI), receiver operating characteristic (ROC) curve and area under the curve (AUC). The definitions of accuracy, precision, recall and F1 score are as follows: Accuracy = (TP+TN)/Total Samples, Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2*(Precision*Recall)/(Precision+Recall). Since the benchmark data set is not balanced, F1 score may be a more substantial evaluation metric.

Citation

@misc{chowdhury2020ecovnet,
  title={ECOVNet: An Ensemble of Deep Convolutional Neural Networks Based on EfficientNet to Detect COVID-19 From Chest X-rays},
  author={Chowdhury et al. (2020)},
  year={2020},
  note={arXiv:2009.11850}
}

• arXiv: 2009.11850