Mlperf Abfp Eval

mlperf-abfp-eval

Adaptive Block Floating-Point for Analog Deep Learning Hardware — Basumallik et al. (2022) (arXiv:2205.06287, 2022)

What this evaluates

Evaluates the inference accuracy of deep neural networks when simulated with Adaptive Block Floating-Point (ABFP) number representation and analog-to-digital converter (ADC) noise. It probes how tile width, amplification gain, and bitwidth affect model quality, and compares the effectiveness of Quantization-Aware Training (QAT) versus Differential Noise Finetuning (DNF) in recovering baseline float32 performance.

Datasets

• MLPerf datacenter inference benchmark — total ?; splits: test (-1)

Metrics

• top-1 accuracy (primary) — range: percent
  • Fraction of correctly classified images out of the total number of images in the dataset.
• mAP — range: percent
  • Mean Average Precision across all object classes, computed as the area under the precision-recall curve.

Input / output format

Input: Pre-trained PyTorch checkpoints with bfloat16 input tensors. Convolutions are converted to tiled matrix-multiplications via im2col; other layers (batch-norm, pooling, non-linearities) compute in float32 before converting back to bfloat16.

Output: Model predictions (class probabilities or bounding boxes) in bfloat16 format, evaluated against ground-truth labels.

Scoring recipe

# For classification (ResNet50)
correct = sum(pred == true_label for pred, true_label in predictions)
accuracy = correct / len(predictions) * 100

# For detection (SSD-ResNet34)
mAP = compute_mean_average_precision(predictions, ground_truth) # Standard MLPerf protocol

Common pitfalls

• Higher amplification gain does not universally improve quality; it can increase noise standard deviation and cause accuracy drops depending on tile width and layer sensitivity.
• Reducing weight/input bitwidths from 8 to 6 bits has negligible impact on quality compared to changes in tile width or gain.
• DNF speed and accuracy depend heavily on selectively injecting noise only into high-variance layers, unlike vanilla QAT which applies noise globally.

Evidence (verbatim from paper)

Table III: Comparing QAT and DNF on ResNet50 and SSD-ResNet34. Bold values highlight methods that improved the DNN to above 99% of the original float32 metric. The metrics are top-1 accuracy (ResNet50) and mAP (SSD-ResNet34).

Citation

@misc{basumallik2022adaptive,
  title={Adaptive Block Floating-Point for Analog Deep Learning Hardware},
  author={Basumallik et al. (2022)},
  year={2022},
  note={arXiv:2205.06287}
}

• arXiv: 2205.06287