Model Serving Latency Eval

model-serving-latency-eval

On the Cost of Model-Serving Frameworks: An Experimental Evaluation — De Rosa et al. (2024) (arXiv:2411.10337, 2024)

What this evaluates

Evaluates the inference latency and end-to-end turn-around time of five machine learning model-serving frameworks across four distinct real-world inference scenarios. It probes how framework specialization (DL-specific vs. general-purpose) and input payload size affect serving performance and stability.

Datasets

• Malware detection — total ?; splits: test (-1)
• Cryptocoin price forecasting — total ?; splits: test (-1)
• Image classification — total ?; splits: test (-1)
• Sentiment analysis — total ?; splits: test (-1)

Metrics

• average_latency (primary) — range: seconds
  • Measured in seconds, reporting both the minimum average latency across payload sizes and the maximum latency at the 99th percentile. Calculated as the mean time difference between request submission and response receipt for each inference call.
• turn_around_time — range: seconds
  • End-to-end time from receiving a raw user HTTP request to returning post-processed predictions in human-readable format, including preprocessing and postprocessing overhead.

Input / output format

Input: Raw unprocessed data (APK files, numerical sequences, JPEG images, or text strings) categorized into small, medium, and large/high payloads. These are preprocessed into tensors before being sent to the serving framework API.

Output: Post-processed predictions in human-readable format (e.g., classification labels or class probabilities) returned via HTTP response.

Scoring recipe

latencies = []
for request in requests:
    start = time.time()
    response = framework.predict(preprocessed_input)
    end = time.time()
    latencies.append(end - start)
avg_latency = sum(latencies) / len(latencies)
p99_latency = np.percentile(latencies, 99)
return {'avg_latency': avg_latency, 'p99_latency': p99_latency}

Common pitfalls

• Confusing raw inference latency with end-to-end turn-around time, which includes Flask-based preprocessing and postprocessing overhead.
• Ignoring payload size variations, as latency and stability (especially at the 99th percentile) can shift significantly with medium/large inputs due to outliers.
• Overlooking framework-model alignment bias, as TensorFlow Serving was tested with native TensorFlow models while TorchServe required model conversion, potentially skewing results.

Evidence (verbatim from paper)

The minimum average latency (among the two frameworks for the three payloads) was 0.0372s, that is 68.37% lower than the one observed for the three general-purpose frameworks (0.1176s); on the other hand, the maximum average latency was 0.0895s, 72.04% lower than the benchmark general-purpose platforms (0.3201s). TensorFlow Serving showed minimum inference times of, respectively, 0.0215s (small payload), 0.0260s (medium payload) and 0.0242s (high payload), and with maximum latencies (at the 99th percentile) of 0.0431s, 0.0445s, 0.0432s.

Citation

@misc{derosa2024modelserving,
  title={On the Cost of Model-Serving Frameworks: An Experimental Evaluation},
  author={De Rosa et al. (2024)},
  year={2024},
  note={arXiv:2411.10337}
}

• arXiv: 2411.10337