Bnn Nvm Benchmark Eval

bnn-nvm-benchmark-eval

Comprehensive Benchmarking of Binary Neural Networks on NVM Crossbar Architectures — Huang et al. (2023) (arXiv:2308.06227, 2023)

What this evaluates

This benchmark evaluates the inference accuracy and hardware performance of binary neural networks (BNNs) deployed on non-volatile memory crossbar architectures. It probes how hardware constraints like ADC resolution and first-layer input precision affect model accuracy, latency, energy efficiency, and chip area.

Datasets

• ImageNet — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Top-1 classification accuracy computed on the ImageNet validation set after training with binary weights and activations.
• latency — range: other
  • Inference latency per layer and end-to-end, measured in nanoseconds via the NeuroSim simulator in pipelined mode.
• energy consumption — range: other
  • Total energy used for inference, measured in picojoules, accounting for ADC resolution and memory cell operations.
• throughput — range: other
  • Number of inference operations completed per second, derived from latency and pipelining parameters.
• chip area — range: other
  • Physical area occupied by the NVM crossbar and peripheral circuitry, measured in square micrometers.

Input / output format

Input: Layer-level model structures (input activation/weight dimensions, pooling/activation parameters) and intermediate activation/weight data from trained BNN models, processed through a NeuroSim PyTorch wrapper.

Output: Simulated hardware metrics (chip area, latency, energy consumption, throughput, efficiency) and inference accuracy scores for each model and ADC resolution configuration.

Scoring recipe

def compute_metrics(predictions, gold_labels, sim_metrics):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = correct / len(gold_labels)
    latency = sim_metrics['latency']
    energy = sim_metrics['energy_consumption']
    throughput = sim_metrics['throughput']
    area = sim_metrics['chip_area']
    return {'accuracy': accuracy, 'latency': latency,
            'energy_consumption': energy, 'throughput': throughput, 'chip_area': area}

Common pitfalls

• NeuroSim defaults to high-precision floating-point input for the first layer; failing to add a bit-serialization module will yield unrealistic accuracy results for BNNs.
• Hardware metrics scale exponentially with ADC resolution and are highly sensitive to first-layer input precision, making direct comparisons across different ADC settings without normalization misleading.

Evidence (verbatim from paper)

To demonstrate the impact of ADC resolution on both accuracy and hardware performance, we perform tests on each model using various ADC resolutions and collect their hardware performance data, including chip area, latency, energy consumption, throughput, and efficiency.

Citation

@misc{huang2023bnnnvm,
  title={Comprehensive Benchmarking of Binary Neural Networks on NVM Crossbar Architectures},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2308.06227}
}

• arXiv: 2308.06227