Svhn Eval

svhn-eval

Fast convolutional neural networks on FPGAs with hls4ml — Aarrestad et al. (2021) (arXiv:2101.05108, 2021)

What this evaluates

Evaluates a model's ability to classify real-world, cropped street-view house numbers into ten digit classes. It probes robustness to natural scene variations such as background clutter, varying colors, orientations, and focus, which are absent in synthetic datasets like MNIST.

Datasets

• Street View House Numbers (SVHN) — total 630420; splits: train (604388), test (26032)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels out of the total number of test images. Reported as the mean across 10 training folds with standard deviation for uncertainty.

Input / output format

Input: 32x32 RGB images normalized to [0, 1] and standardized to zero mean and unit variance per pixel.

Output: Single integer class label from 0 to 9.

Scoring recipe

correct = 0
for img, label in test_set:
    pred = model(img)
    if pred == label:
        correct += 1
accuracy = correct / len(test_set)

Common pitfalls

• Ground truth labels correspond strictly to the center digit when multiple digits appear in an image.
• Performance metrics are averaged across 10 training folds rather than reported on a single fixed train/test split.

Evidence (verbatim from paper)

Model-performance figures of merit (e.g., accuracy, true and false positive rates, etc.) are defined considering the mean across the 10 folds on the test set.

Citation

@misc{aarrestad2021fast,
  title={Fast convolutional neural networks on FPGAs with hls4ml},
  author={Aarrestad et al. (2021)},
  year={2021},
  note={arXiv:2101.05108}
}

• arXiv: 2101.05108