Olive Branch Learning Eval

olive-branch-learning-eval

Olive Branch Learning: A Topology-Aware Federated Learning Framework for Space-Air-Ground Integrated Network — Fang et al. (2022) (arXiv:2212.01215, 2022)

What this evaluates

Evaluates a topology-aware federated learning framework (OBL) with a satellite-assignment algorithm (CNASA) for space-air-ground integrated networks. It probes the trade-off between model accuracy and training latency under strict non-IID data distributions and varying network topologies.

Datasets

• MNIST — total ?; splits: train (-1), test (-1); HF mnist
• Fashion-MNIST — total ?; splits: train (-1), test (-1); HF fashion-mnist
• CIFAR-10 — total ?; splits: train (-1), test (-1); HF cifar10

Metrics

• final global model accuracy (primary) — range: percent
  • Percentage of correctly classified samples on the held-out test set after the final global training round.
• overall time cost of model training — range: other
  • Total wall-clock time required to complete all global federated learning rounds, including local computation, intra-cluster aggregation, and inter-satellite communication delays.

Input / output format

Input: Local image classification datasets (MNIST/Fashion-MNIST/CIFAR-10) partitioned non-IIDly across 200 IoRT devices (each holding exactly 2 classes). Models are CNNs trained locally.

Output: Global aggregated model parameters after each round, evaluated on a held-out test set to yield test accuracy and total training time.

Scoring recipe

def compute_metrics(predictions, gold_labels, total_time):
    correct = (predictions.argmax(dim=1) == gold_labels).sum().item()
    accuracy = correct / len(gold_labels) * 100
    return {'final global model accuracy': accuracy, 'overall time cost of model training': total_time}

Common pitfalls

• The non-IID setting is highly constrained: each of the 200 devices holds data from exactly 2 classes, and geographically close devices share similar class distributions.
• Time cost is not just computation; it explicitly includes propagation delays (5 ms device-to-air, 5 ms air-to-satellite, 20 ms satellite-to-satellite) and bandwidth constraints (6000 Mbps satellite, 32 Gbps air node), which vary by topology.
• Baseline methods (GDO, CDO) are custom adaptations of FedAT and Astraea for SAGIN topologies, not standard FedAvg or SCAFFOLD implementations.

Evidence (verbatim from paper)

Our evaluation metrics include overall time cost of model training and final global model accuracy. ... Each dataset consists of a training set for model training and a test set for model evaluation... To mimic the challenging non-IID scenario, we assign each IoRT device samples of 2 classes, which will bring a significant negative impact on the global model accuracy.

Citation

@misc{fang2022olivebranch,
  title={Olive Branch Learning: A Topology-Aware Federated Learning Framework for Space-Air-Ground Integrated Network},
  author={Fang et al. (2022)},
  year={2022},
  note={arXiv:2212.01215}
}

• arXiv: 2212.01215