Crowdsensing Id Dfl Eval

crowdsensing-id-dfl-eval

A Crowdsensing Intrusion Detection Dataset For Decentralized Federated Learning Models — Feng et al. (2025) (arXiv:2507.13313, 2025)

What this evaluates

Evaluates the capability of decentralized federated learning (DFL) models to detect malware and classify benign states in IoT crowdsensing environments. It probes robustness under varying node counts, peer-to-peer network topologies, and data heterogeneity (IID vs. non-IID Dirichlet splits).

Datasets

• Crowdsensing Intrusion Detection Dataset — total 342000; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: fraction of correctly predicted instances out of total instances.
• f1_score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
• precision — range: [0, 1]
  • True positives divided by the sum of true and false positives.
• recall — range: [0, 1]
  • True positives divided by the sum of true positives and false negatives.

Input / output format

Input: 30-second aggregated behavioral feature vectors capturing system calls, file system operations, I/O, network activity, resource usage, and kernel events from IoT devices.

Output: Nine-class classification label (1 benign class + 8 malware families).

Scoring recipe

from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score

def compute_metrics(y_true, y_pred):
    acc = accuracy_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred, average='macro')
    prec = precision_score(y_true, y_pred, average='macro')
    rec = recall_score(y_true, y_pred, average='macro')
    return acc, f1, prec, rec

Common pitfalls

• Non-IID data distribution (Dirichlet α=1) significantly degrades performance, which is often conflated with algorithmic failure rather than dataset heterogeneity.
• Increasing the number of federated nodes reduces per-node data volume, causing expected performance drops that are dataset-specific and not solely due to communication overhead.
• Network topology (fully connected, ring, star) shows minimal impact on this dataset, contrary to assumptions that topology heavily dictates DFL convergence.

Evidence (verbatim from paper)

The evaluation metrics used in all experiments include Accuracy, F1 score, Precision, and Recall, which collectively measure classification performance from multiple perspectives.

Citation

@misc{feng2025crowdsensing,
  title={A Crowdsensing Intrusion Detection Dataset For Decentralized Federated Learning Models},
  author={Feng et al. (2025)},
  year={2025},
  note={arXiv:2507.13313}
}

• arXiv: 2507.13313