Fed Echo Eval

fed-echo-eval

FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data — Zhang et al. (2024) (arXiv:2411.07050, 2024)

What this evaluates

Evaluates federated learning algorithms on echocardiogram video segmentation tasks under label incompleteness and high data heterogeneity across multiple medical institutions. It probes how FL methods handle missing annotations and conflicting labels from different clinical sites.

Datasets

• Fed-ECHO — total ?; splits: train (-1), test (-1); repo https://github.com/SMILELab-FL/FedCVD

Metrics

• DICE (primary) — range: percent
  • Dice similarity coefficient measuring overlap between predicted and ground truth segmentation masks, expressed as a percentage.
• Hausdorff distance ($d_H$) — range: other
  • Maximum distance between the boundaries of the predicted and ground truth segmentation masks.

Input / output format

Input: Echocardiogram video frames from a specific medical institution (client).

Output: Segmentation masks for cardiac regions (BG, LV_Endo, LV_Epi, LA).

Scoring recipe

def compute_dice(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    union = np.sum(pred_mask | gt_mask)
    return 2 * intersection / union * 100 if union > 0 else 0.0
def compute_hd(pred_mask, gt_mask):
    pred_pts = np.argwhere(pred_mask)
    gt_pts = np.argwhere(gt_mask)
    if len(pred_pts) == 0 or len(gt_pts) == 0: return float('inf')
    dists = cdist(pred_pts, gt_pts)
    return np.max(np.min(dists, axis=1)) + np.max(np.min(dists, axis=0))

Common pitfalls

• Label incompleteness means some institutions only annotate specific regions; 'Maybe-BG' labels from other institutions must be masked during loss calculation and evaluation.
• Evaluating on global test set without masking unreliable labels inflates error metrics.
• Semi-supervised baselines require extra rounds for labeled client training before FL begins.

Evidence (verbatim from paper)

Table 4: The performance of different FL methods on Fed-ECHO, with DICE (%) and $d_{H}$ representing DICE index and Hausdorff distance respectively.

Citation

@misc{zhang2024fedcvd,
  title={FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2411.07050}
}

• arXiv: 2411.07050