Neuromorphic Cnn Eval

neuromorphic-cnn-eval

Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing — Esser et al. (2016) (arXiv:1603.08270, 2016)

What this evaluates

Evaluates the classification accuracy and energy efficiency of a neuromorphic convolutional network architecture running on Intel's TrueNorth hardware. It probes the model's ability to perform real-time visual and audio recognition while maintaining low power consumption and high throughput.

Datasets

• CIFAR10 — total ?; splits: test (-1)
• CIFAR100 — total ?; splits: test (-1)
• SVHN — total ?; splits: test (-1)
• GTSRB — total ?; splits: test (-1)
• Flickr-Logos32 — total ?; splits: test (-1)
• VAD — total ?; splits: test (-1)
• TIMIT Class. — total ?; splits: test (-1)
• TIMIT Frame — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified inputs out of the total number of test instances.
• FPS/W — range: other
  • Frames per second divided by power consumption in watts. Computed by scaling idle leakage power and active power measurements based on core usage fraction and classification throughput.

Input / output format

Input: 32×32×3 RGB images for visual tasks, or MFCC feature sequences (e.g., 16×26 or 32×16×3) for audio tasks.

Output: Single class label per input instance.

Scoring recipe

accuracy = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)

# FPS/W calculation per paper protocol
idle_power = measure_idle_power_on_NS1t()
active_power = measure_total_power_on_NS1t() - idle_power
scaled_power = idle_power * (used_cores / total_cores) + active_power * (measured_fps / max_fps)
fps_per_watt = measured_fps / (scaled_power / 1000.0)

Common pitfalls

• Data augmentation (e.g., mirroring) and dropout were explicitly excluded during training to isolate algorithm performance, so comparisons must use non-augmented baselines.
• Power measurement requires two separate hardware boards (NS1e for throughput, NS1t for power) and involves scaling idle leakage and active power by core fraction and FPS, which can introduce measurement discrepancies if not replicated exactly.
• Multi-chip networks were evaluated in simulation, while single-chip networks were evaluated on actual hardware, meaning hardware/simulation differences may affect direct accuracy comparisons across chip counts.

Evidence (verbatim from paper)

Our experiments show that for almost all of the benchmarks, a single-chip network is sufficient to come within a few percent of state-of-the-art accuracy. Increasing to up to 8 chips improved accuracy by several percentage points, and in the case of the VAD dataset surpassed state-of-the-art performance.

Citation

@misc{esser2016convolutional,
  title={Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing},
  author={Esser et al. (2016)},
  year={2016},
  note={arXiv:1603.08270}
}

• arXiv: 1603.08270