qhjqhj00 finn-bnn-inference-eval
FINN: A Framework for Fast, Scalable Binarized Neural Network Inference — Umuroglu et al. (2016) (arXiv:1612.07119, 2016)
What this evaluates
Evaluates the inference performance of binarized neural networks (BNNs) accelerated on FPGAs using the FINN framework. It measures classification throughput, latency, and accuracy across standard image datasets to assess hardware efficiency and resource utilization.
Datasets
- MNIST — total ?; splits: test (10000)
- CIFAR-10 — total ?; splits: test (10000)
- SVHN — total ?; splits: test (10000)
Metrics
classification throughput (FPS)(primary) — range: [0, ∞)- FPS = N / T, where N is the number of processed images (10,000) and T is the measured execution time in seconds.
latency (μs)— range: [0, ∞)- Time taken to classify a single image, measured in microseconds.
accuracy (%)— range: [0, 100]- Percentage of correctly classified images out of the total test set.
power consumption (W)— range: [0, ∞)- Measured as P_chip (FPGA internal power via PMBus) and P_wall (total board power via wall meter).
Input / output format
Input: Binary or low-precision images (28x28 for MNIST, 32x32 for CIFAR-10/SVHN) initialized in shared DRAM.
Output: 10-bit one-hot vector (for fully connected topologies) or 10-element 16-bit vector (for convolutional topology) indicating class logits/probabilities.
Scoring recipe
def evaluate(accelerator, test_images, ground_truth):
N = len(test_images)
start_time = time.time()
predictions = accelerator.run(test_images)
end_time = time.time()
fps = N / (end_time - start_time)
accuracy = sum(1 for p, g in zip(predictions, ground_truth) if p == g) / N
return fps, accuracy
Common pitfalls
- Confusing P_chip (FPGA internal power) with P_wall (total board power), as P_wall includes a ~7W idle baseline that dominates low-power measurements.
- Assuming higher folding factors linearly improve throughput; they are constrained by integer folding and BRAM/LUT limits, often causing resource bottlenecks or unused capacity.
- Comparing FPS across different datasets without accounting for varying image resolutions and network complexities.
Evidence (verbatim from paper)
The host code runs on the Cortex-A9 cores of the Zynq. It initializes 10000 images with test data in the Zynq's shared DRAM, launches and times the accelerator execution to measure classification throughput, then measures accuracy by comparing against the correct classifications.
Citation
@misc{umuroglu2016finn,
title={FINN: A Framework for Fast, Scalable Binarized Neural Network Inference},
author={Umuroglu et al. (2016)},
year={2016},
note={arXiv:1612.07119}
}
- arXiv: 1612.07119