Lazybatching Eval

lazybatching-eval

LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference — Choi, Kim, and Rhu (2020) (arXiv:2010.13103, 2020)

What this evaluates

Evaluates the inference latency, throughput, and SLA compliance of a dynamic batching system under varying request arrival rates and diverse DNN workloads.

Datasets

• ResNet — total ?; splits: test (-1)
• GNMT — total ?; splits: test (-1)
• Transformer — total ?; splits: test (-1)
• VGGNet — total ?; splits: test (-1)
• MobileNet — total ?; splits: test (-1)
• LAS — total ?; splits: test (-1)
• BERT — total ?; splits: test (-1)

Metrics

• SLA violation rate (primary) — range: [0, 1]
  • Fraction of inference requests where end-to-end latency exceeds the specified SLA deadline target.
• average latency — range: ms
  • Mean end-to-end inference time per query across all requests.
• throughput — range: req/sec
  • Number of inference queries processed per second.

Input / output format

Input: Simulated inference requests arriving at a specified query-arrival rate (requests/sec) against target DNN models.

Output: Per-request end-to-end inference latency, system throughput, and SLA compliance status.

Scoring recipe

def compute_sla_violation_rate(latencies, sla_target_ms):
    violations = sum(1 for lat in latencies if lat > sla_target_ms)
    return violations / len(latencies)

Common pitfalls

• SLA deadlines are vendor-specific and not publicly disclosed; the paper assumes a 100 ms default for slack estimation.
• Results are primarily based on simulations (20 runs) rather than real hardware traces for the primary NPU setup.
• The dec_timesteps parameter for dynamic DNN latency estimation significantly impacts SLA violation rates if set too low.

Evidence (verbatim from paper)

To quantify how well our LazyBatching scheduler minimizes SLA violations, we sweep the SLA target value (SLA_target in Equation 2) and measure the fraction of SLA violated inference requests as a function of different batching policies. As shown in Figure 15, graph batching experiences severe SLA violations even when the SLA target is set up loosely (e.g., even at SLA target of 100 ms, two-thirds of graph batching configurations experience more than 50% violations).

Citation

@misc{choi2020lazybatching,
  title={LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inference},
  author={Choi, Kim, and Rhu (2020)},
  year={2020},
  note={arXiv:2010.13103}
}

• arXiv: 2010.13103