Finn R Eval

finn-r-eval

FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks — Blott et al. (2018) (arXiv:1809.04570, 2018)

What this evaluates

Evaluates the performance, power, and resource efficiency of quantized neural networks deployed on various FPGA platforms using the FINN-R framework. It probes the trade-offs between network precision, hardware resource usage, throughput, and classification accuracy across embedded and datacenter-scale hardware.

Datasets

• MNIST — total ?; splits: test (-1)
• CIFAR-10 — total ?; splits: test (-1)
• GTSRB — total ?; splits: test (-1)
• SVHN — total ?; splits: test (-1)
• VOC 2007 — total ?; splits: test (-1)
• ImageNet — total ?; splits: test (-1)

Metrics

• Top-1 Accuracy (primary) — range: percent
  • Percentage of correctly classified images out of the total test set.
• mAP — range: percent
  • Mean Average Precision for object detection tasks, averaging precision across all recall levels.
• Performance — range: GOp/s
  • Total additions and multiplications per second computed as input frames pass through convolutional layers and perceptrons.
• Power — range: W
  • Board-level power consumption measured in watts.
• Efficiency — range: GOp/s/W
  • Performance divided by power consumption.

Input / output format

Input: Image frames passed through convolutional layers and perceptrons.

Output: Classification labels or detection boxes, plus hardware metrics (clock frequency, BRAM/LUT usage, performance in GOp/s, power in W, efficiency in GOp/s/W).

Scoring recipe

def evaluate(qnn_model, test_loader, power_w):
    correct = 0
    total = 0
    ops_per_frame = count_dot_products(qnn_model)
    fps = measure_throughput(qnn_model, test_loader)
    for images, labels in test_loader:
        preds = qnn_model(images)
        correct += (preds.argmax(1) == labels).sum().item()
        total += labels.size(0)
    accuracy = correct / total
    performance_gops = (ops_per_frame * fps) / 1e9
    efficiency = performance_gops / power_w
    return accuracy, performance_gops, efficiency

Common pitfalls

• Board-level power measurements are only available for embedded platforms (Ultra96, PYNQ-Z1); cloud platforms (AWS F1) lack power data, making cross-platform efficiency comparisons invalid.
• Performance numbers for prior work are often estimated or extrapolated, and the authors explicitly exclude extrapolated numbers for their own comparison but note that their own results lack full timing closure optimization.
• Resource counts (BRAM, LUTs) are only reported for Xilinx devices and may exclude URAM/DSP depending on precision layers.

Evidence (verbatim from paper)

The stated figures sum up all the additions and multiplications within the dot products computed as an individual input frames passes through the convolutional layers and perceptrons of a topology. All implementations are compared in regards to performance, power, efficiency (=performance/power) for a specific network on a specific platform.

Citation

@misc{blott2018finnr,
  title={FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks},
  author={Blott et al. (2018)},
  year={2018},
  note={arXiv:1809.04570}
}

• arXiv: 1809.04570