Edgesnn Evaluation Protocol

edgesnn-evaluation-protocol

Edge Intelligence with Spiking Neural Networks — Deng et al. (2025) (arXiv:2507.14069, 2025)

What this evaluates

Evaluates the performance, efficiency, and robustness of Spiking Neural Networks (SNNs) deployed on edge hardware or simulated on conventional processors. It probes hardware-independent algorithmic complexity and system-level execution metrics under resource-constrained, latency-sensitive conditions.

Datasets

• (no dataset; pure metric skill)

Metrics

• accuracy / mAP / MSE (primary) — range: [0, 1]
  • Standard task-specific correctness metrics. Accuracy is the proportion of correct predictions; mAP is mean average precision for detection/segmentation; MSE is mean squared error for regression.
• footprint — range: other
  • Theoretical memory in bytes required to represent the model, decomposable into synaptic weight count, trainable neuronal parameters, and data buffer requirements.
• sparsity — range: [0, 1]
  • Connection sparsity: ratio of zero weights to total weights (0 to 1). Activation sparsity: average proportion of inactive neuron outputs across neurons, layers, time steps, and test samples (0 to 1).
• synaptic_operations — range: other
  • Average number of synaptic operations (SOPs) per execution. Categorized into dense SOPs (all operations), MACs (non-binary activations/weights), and ACs (spike-based operations).
• latency — range: other
  • Execution time per inference in a streaming scenario where each inference starts only after the previous one finishes.
• energy_efficiency — range: other
  • Energy consumption or average/peak power per inference or per task, often estimated by combining effective SOPs with chip-specific energy parameters.
• resilience — range: other
  • System dependability metrics: Recovery Time Objective (RTO), Mean Time to Repair (MTTR), and Mean Time Between Failures (MTBF).

Input / output format

Input: SNN model architecture and task-specific workload data (e.g., image, video, or temporal sequences) for simulation or hardware deployment.

Output: Task predictions (labels, bounding boxes, or continuous values) and system execution logs (timing, power, memory usage).

Scoring recipe

def compute_metrics(predictions, gold, model_config, execution_log):
    # Correctness
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    # Complexity (static)
    footprint = model_config.total_params * 4  # bytes
    sparsity = model_config.zero_weights / model_config.total_weights
    # Complexity (dynamic/workload-dependent)
    sop = execution_log.synaptic_ops
    latency = execution_log.streaming_inference_time
    # Efficiency
    energy = sop * execution_log.energy_per_sop
    return {'accuracy': acc, 'footprint': footprint, 'sparsity': sparsity, 'sop': sop, 'latency': latency, 'energy': energy}

Common pitfalls

• Confusing hardware-simulated SOPs with actual chip energy consumption without accounting for software stack overhead and real-world I/O latency.
• Reporting only static metrics (footprint, connection sparsity) while ignoring workload-dependent dynamic metrics (activation sparsity, latency, energy).
• Using batched offline throughput metrics for real-time streaming edge tasks, which misrepresents actual latency requirements.

Evidence (verbatim from paper)

The algorithmic track defines platform-agnostic primary metrics that are broadly applicable to diverse EdgeSNN solutions. These include: (1) Correctness metrics, which evaluate the quality of model predictions for specific tasks, such as accuracy, mean average precision (mAP), and mean squared error (MSE), and (2) Complexity metrics, which assess the theoretical computational demands of the algorithm. ... Timing performance encompasses sample throughput and execution time (i.e., latency). ... efficiency metrics should be benchmark-specific—for example, average power for always-on tasks, and peak power or energy cost per inference for high-throughput workloads.

Citation

@misc{deng2025edgesnn,
  title={Edge Intelligence with Spiking Neural Networks},
  author={Deng et al. (2025)},
  year={2025},
  note={arXiv:2507.14069}
}

• arXiv: 2507.14069