Fedaiot Eval

fedaiot-eval

FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things — Samiul Alam et al. (2023) (arXiv:2310.00109, 2023)

What this evaluates

Evaluates federated learning algorithms on authentic IoT data modalities under realistic constraints like non-IID data partitioning, label noise, and quantized training. It probes how data heterogeneity, client sampling ratios, and hardware limitations affect model convergence and final performance across diverse sensing tasks.

Datasets

• WISDM-W — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
• WISDM-P — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
• UT-HAR — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
• Widar — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
• VisDrone — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
• CASAS — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
• AEP — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT
• EPIC-SOUNDS — total ?; splits: (unstated); repo https://github.com/AIoT-MLSys-Lab/FedAIoT

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly classified instances out of the total number of instances. Calculated as (TP + TN) / (TP + TN + FP + FN) * 100.
• MAP-50 (%) — range: percent
  • Mean Average Precision at an Intersection over Union (IoU) threshold of 0.5. Averages the precision-recall curve for each class at the specified IoU threshold.
• R² — range: [0, 1]
  • Coefficient of determination measuring the proportion of variance in the dependent variable predictable from the independent variables. Calculated as 1 - (SS_res / SS_tot).

Input / output format

Input: IoT sensor data (accelerometer, gyroscope, Wi-Fi CSI, images, audio, smart home logs) partitioned across multiple clients with non-IID distributions. Clients receive local data batches for training and send model updates to a central server.

Output: Predicted class labels or regression values per client/model update. Final evaluation reports mean and standard deviation over three random seeds.

Scoring recipe

def compute_accuracy(preds, gold):
    return (preds == gold).mean() * 100

def compute_map50(preds, gold, iou_thresh=0.5):
    # Standard object detection MAP calculation at IoU=0.5
    return mean_average_precision(preds, gold, iou_thresh=iou_thresh) * 100

def compute_r2(preds, gold):
    ss_res = np.sum((gold - preds) ** 2)
    ss_tot = np.sum((gold - np.mean(gold)) ** 2)
    return 1 - (ss_res / ss_tot)

Common pitfalls

• Data heterogeneity level (alpha) impacts performance differently across datasets; some are extremely sensitive while others show limited differences.
• Quantized training (FP16) performance varies significantly by dataset; some modalities suffer drastic accuracy drops while others improve compared to FP32.
• Label noise sensitivity is highly dataset-dependent; some models drop significantly at just 10% noise while others remain stable.

Evidence (verbatim from paper)

Table 5: Overall performance. ... Metric: Accuracy (%) ... Table 8: Performance on quantized training. ... Metric: Accuracy (%) ... MAP-50 (%) ... R²

Citation

@misc{alam2023fedaiot,
  title={FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Things},
  author={Samiul Alam et al. (2023)},
  year={2023},
  note={arXiv:2310.00109}
}

• arXiv: 2310.00109