Fpga Inference Latency Throughput

fpga-inference-latency-throughput

FPGA-accelerated machine learning inference as a service for particle physics computing — Duarte et al. (2019) (arXiv:1904.08986, 2019)

What this evaluates

Evaluates the end-to-end system performance of an FPGA-accelerated machine learning inference service, specifically measuring inference latency and throughput under varying network conditions and concurrent workloads.

Datasets

• Simulated CMS dataset (jet images) — total ?; splits: (unstated); repo https://github.com/hls-fpga-machine-learning/SonicCMS

Metrics

• round-trip inference latency (primary) — range: ms
  • Total time from transmission of the input image to receipt of the classification result, measured in milliseconds. Reported as mean, median, and distribution.
• throughput — range: ips
  • Number of inferences completed per second, calculated as total inferences divided by total processing time for concurrent processes.
• failure_rate — range: percent
  • Percentage of inference requests that time out during the test.

Input / output format

Input: 2D grayscale image tensor representing a reconstructed particle jet, fed into a ResNet-50 model.

Output: Classification result (e.g., jet tag or event type).

Scoring recipe

start = time.time()
pred = model.predict(image)
latency_ms = (time.time() - start) * 1000
total_time = max(end - start for concurrent jobs)
throughput = (5000 * num_jobs) / total_time
failures = count(requests exceeding timeout)
failure_rate = failures / (5000 * num_jobs)

Common pitfalls

• Network transmission time dominates remote latency (up to 60 ms), while FPGA inference itself is only ~1.8 ms.
• Throughput scales with concurrent CPU processes but plateaus at ~650 ips due to the FPGA pipeline limit, not linearly with process count.
• Remote latency is heavily dependent on physical distance and network conditions, with a hard physical limit of ~10 ms round-trip to Azure East 2.

Evidence (verbatim from paper)

We measure the total round-trip latency of the inference request as seen by CMSSW, starting from the transmission of the image and ending with the receipt of the classification results. ... We also measure the throughput based on the total time for each simultaneous process to complete serial processing of 5000 jet images.

Citation

@misc{duarte2019fpga,
  title={FPGA-accelerated machine learning inference as a service for particle physics computing},
  author={Duarte et al. (2019)},
  year={2019},
  note={arXiv:1904.08986}
}

• arXiv: 1904.08986