Dnn Inference Latency Eval

dnn-inference-latency-eval

Benchmarking Different Application Types across Heterogeneous Cloud Compute Services — Duggi et al. (2025) (arXiv:2501.06128, 2025)

What this evaluates

Probes the inference latency and execution time variability of four industrial DNN models across heterogeneous cloud compute instances (CPU, GPU, and inference-optimized) on AWS and Chameleon Cloud. It evaluates how hardware heterogeneity impacts real-time processing requirements for safety-critical industrial applications.

Datasets

• Fire Detection Dataset — total 240; splits: test (-1)
• Oil Spill Detection Dataset — total 110; splits: test (-1)
• UCI Human Activity Recognition Using Smartphones — total ?; splits: test (-1)
• Marmousi 2 Dataset — total ?; splits: test (-1)

Metrics

• inference_time (primary) — range: other
  • Measured as the wall-clock execution time for a single inference pass. Aggregated across multiple runs using mean and standard deviation to quantify performance and variability.

Input / output format

Input: Model-specific input data: video frames for FireNet, SAR images for FCN-8, sequential motion sensor data for HAR, and seismic data arrays for the TCN model.

Output: Model predictions (e.g., segmentation masks, activity class labels, or impedance estimates) alongside the recorded wall-clock execution time.

Scoring recipe

times = []
for _ in range(num_runs):
    start = time.perf_counter()
    _ = model(input_data)
    end = time.perf_counter()
    times.append(end - start)
mean_time = sum(times) / len(times)
std_time = (sum((t - mean_time)**2 for t in times) / len(times)) ** 0.5
return {"mean_inference_time": mean_time, "std_inference_time": std_time}

Common pitfalls

• Failing to run multiple iterations per instance, which ignores the stochastic nature of cloud inference latency and produces misleading single-run measurements.
• Comparing raw latency values across different instance types (CPU vs GPU vs inf1) without accounting for hardware specifications or workload characteristics.
• Overlooking statistical distribution testing (Shapiro-Wilk/Kolmogorov-Smirnov) required to properly model execution time variability for safety-critical deployments.

Evidence (verbatim from paper)

To benchmark the DNN applications, the inference time is measured in AWS and Chameleon cloud providers with respect to different machine types they offer. That is, the aforementioned applications are deployed across different cloud instances mentioned earlier. The random nature of the inference time is captured by running each application multiple times under varying cloud conditions to obtain a comprehensive understanding of how the system's heterogeneity influences the performance. ... Mean and Standard Deviation of Inference Executive Times The mean and standard deviation of the inference times summarizes the behavior of the observations in a single value.

Citation

@misc{duggi2025benchmarking,
  title={Benchmarking Different Application Types across Heterogeneous Cloud Compute Services},
  author={Duggi et al. (2025)},
  year={2025},
  note={arXiv:2501.06128}
}

• arXiv: 2501.06128