Etree Edge AI Eval

etree-edge-ai-eval

E-Tree Learning: A Novel Decentralized Model Learning Framework for Edge AI — Yang et al. (2020) (arXiv:2008.01553, 2020)

What this evaluates

Evaluates decentralized model aggregation frameworks on edge devices under IID and Non-IID data distributions. It measures classification accuracy and convergence speed to compare hierarchical tree-based learning against centralized federated learning and fully decentralized gossip learning.

Datasets

• HAR Using Smartphones Dataset — total 10299; splits: train (7352), test (2947)
• Pendigits — total ?; splits: train (-1), test (-1)

Metrics

• classification accuracy (primary) — range: percent
  • Proportion of correctly classified test samples out of the total test set size.
• convergence time — range: other
  • Wall-clock time or number of aggregation rounds required to reach a stable accuracy threshold.

Input / output format

Input: Feature vectors (e.g., 561 features for HAR) distributed across edge devices, processed via local training and hierarchical/centralized aggregation.

Output: Aggregated model parameters for the next round, or final class label predictions evaluated on the test set.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def check_convergence(accuracy_history, threshold=0.001):
    for i in range(1, len(accuracy_history)):
        if abs(accuracy_history[i] - accuracy_history[i-1]) < threshold:
            return i
    return len(accuracy_history)

Common pitfalls

• Non-IID distribution assigns exactly 4 out of 6 classes per device, not just volume imbalance.
• Gossip learning reports average accuracy across all local models rather than a single global model.
• Convergence speed differs between 'rounds' and 'time' due to configurable aggregation frequencies per tree layer.

Evidence (verbatim from paper)

We define two performance metrics. For federated learning and E-Tree learning, we measure the classification accuracy of the model in every round of model aggregation, where a round in E-Tree means the root node finishes an aggregation. The accuracy is calculated using the same testing dataset. Gossip learning does not have a consolidated model after a round of learning. Instead every edge device maintains a separate model. So we measure the average model accuracy of all the edge devices. We concern on the convergency of the methods. We want to compare the final and converged accuracy of the methods.

Citation

@misc{yang2020etree,
  title={E-Tree Learning: A Novel Decentralized Model Learning Framework for Edge AI},
  author={Yang et al. (2020)},
  year={2020},
  note={arXiv:2008.01553}
}

• arXiv: 2008.01553