qhjqhj00 integer-quantization-eval
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference — Benoit Jacob et al. (arXiv:1712.05877, 2017)
What this evaluates
Evaluates the classification and detection accuracy, as well as inference latency, of neural networks quantized to 8-bit integer arithmetic on mobile ARM CPUs compared to floating-point baselines.
Datasets
- ImageNet — total ?; splits: val (-1)
- COCO — total ?; splits: val (-1)
- Face detection dataset — total ?; splits: test (-1)
- Face attributes dataset — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly predicted class labels out of total samples.
mAP— range: percent- Mean Average Precision computed over IoU thresholds from 0.5 to 0.95 for object detection.
latency (ms)— range: ms- Inference time measured in milliseconds on single-threaded or multi-threaded ARM CPU cores.
Input / output format
Input: Preprocessed image inputs fed into quantized or floating-point MobileNet/ResNet/Inception models.
Output: Predicted class labels, bounding boxes, or attribute values, along with measured inference latency in milliseconds.
Scoring recipe
def compute_metrics(predictions, ground_truth, latencies):
correct = sum(1 for p, g in zip(predictions, ground_truth) if p == g)
accuracy = correct / len(ground_truth) * 100
ap_scores = []
for cls in classes:
prec, rec = compute_precision_recall_curve(predictions[cls], ground_truth[cls])
ap_scores.append(interpolate_ap(prec, rec))
mAP = sum(ap_scores) / len(classes)
avg_latency = sum(latencies) / len(latencies)
return accuracy, mAP, avg_latency
Common pitfalls
- Latency is highly dependent on specific ARM micro-architectures (Snapdragon 835/821) and thread counts, making cross-device comparisons difficult.
- Accuracy is often reported as relative degradation from the floating-point baseline rather than absolute values.
- Quantization bit-depths for weights and activations are ablated independently, which can obscure total model size vs. accuracy tradeoffs.
Evidence (verbatim from paper)
Integer-only quantized MobileNets achieve higher accuracies than floating-point MobileNets given the same run time budget. The accuracy gap is quite substantial (~10%) for Snapdragon 835 LITTLE cores at the 33ms latency needed for real-time (30 fps) operation.
Citation
@misc{jacob2017quantization,
title={Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference},
author={Benoit Jacob et al.},
year={2017},
note={arXiv:1712.05877}
}
- arXiv: 1712.05877