Tf2aif Inference Latency Eval

tf2aif-inference-latency-eval

TF2AIF: Facilitating development and deployment of accelerated AI models on the cloud-edge continuum — Leftheriotis et al. (2024) (arXiv:2404.13715, 2024)

What this evaluates

Evaluates the execution latency and code-generation efficiency of automated, platform-specific AI inference engines across heterogeneous cloud-edge hardware (CPU, GPU, ARM, FPGA, SoC) for image classification tasks.

Datasets

• Image classification dataset (unspecified) — total ?; splits: test (-1)

Metrics

• execution latency (primary) — range: milliseconds
  • Average time in milliseconds to process a single inference request over 1000 trials per model-platform variant.
• generation time — range: seconds
  • Time in seconds to compose Docker images and convert TensorFlow models to platform-specific inference engines.
• speedup vs native TensorFlow — range: ratio
  • Ratio of average native TensorFlow latency to average TF2AIF-generated framework latency for the same model and platform.

Input / output format

Input: Single image per inference request.

Output: Inference result and execution timestamp for latency measurement.

Scoring recipe

latencies = []
for variant in variants:
    for _ in range(1000):
        start = time.time()
        _ = send_request(variant, image)
        end = time.time()
        latencies.append((end - start) * 1000)
    avg_latency = sum(latencies) / len(latencies)
    speedup = native_tf_latency / avg_latency

Common pitfalls

• CPU latency shows high variability due to system noise like context switching, requiring careful benchmarking conditions.
• FPGA (ALVEO) conversion time is significantly higher than other platforms due to Vitis-AI toolchain overhead.
• Native TensorFlow comparison excludes FPGA platforms due to lack of native support, making cross-platform speedup comparisons incomplete for FPGAs.

Evidence (verbatim from paper)

For our performance benchmarking, we conducted 1000 requests for each AI-framework-platform model variant, focusing on the execution latency measured in milliseconds.

Citation

@misc{leftheriotis2024tf2aif,
  title={TF2AIF: Facilitating development and deployment of accelerated AI models on the cloud-edge continuum},
  author={Leftheriotis et al. (2024)},
  year={2024},
  note={arXiv:2404.13715}
}

• arXiv: 2404.13715