LLM Serving Eval

llm-serving-eval

BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure — He et al. (2025) (arXiv:2510.13223, 2025)

What this evaluates

Evaluates the throughput, latency, and scalability of LLM inference serving systems under varying request rates and context lengths. It probes how efficiently a system manages KV cache, batching, and resource allocation for both short and long-context instruction-following workloads.

Datasets

• Alpaca — total 52000; splits: test (52000)
• LongBench — total ?; splits: test (-1)

Metrics

• Throughput (primary) — range: tokens/s
  • Total number of generated tokens divided by the total wall-clock time from the first request submission to the last response completion. Measured in tokens per second.
• Total processing time — range: seconds
  • End-to-end latency from request submission to complete response generation, capturing all system overheads including model loading, attention computation, KV Cache management, and post-processing.
• Average latency — range: seconds
  • Mean per-request completion time, decomposed into Time-To-First-Token (TTFT) and inter-token latency. TTFT measures responsiveness; inter-token measures generation smoothness.

Input / output format

Input: Instruction/prompt text (Alpaca: 4–50 tokens; LongBench: ~2,000–85,000+ tokens).

Output: Generated text response, capped at a maximum of 512 tokens per request.

Scoring recipe

# 1. Warm up system for 60s
# 2. Generate requests at target RPS using Poisson arrival process
# 3. For each request i: record t_sub[i] (submission) and t_comp[i] (completion)
# 4. total_tokens = sum(len(response_i) for all i)
# 5. throughput = total_tokens / (max(t_comp) - min(t_sub))
# 6. total_time = mean(t_comp[i] - t_sub[i] for all i)
# 7. ttft = mean(time_to_first_token_i for all i)
# 8. inter_token = mean((t_comp[i] - t_sub[i] - ttft_i) / (num_tokens_i - 1))
# 9. Repeat steps 2-8 five times with different random seeds
# 10. Report mean ± 95% confidence interval across runs

Common pitfalls

• Max output length is strictly capped at 512 tokens to ensure comparability; ignoring this cap invalidates throughput and latency comparisons.
• Poisson arrival process is used for request generation, not uniform; using uniform arrivals will misrepresent queueing behavior and system resilience under load.
• Metrics are reported as mean values with 95% confidence intervals over 5 runs; single-run results are statistically invalid and should not be used for comparison.

Evidence (verbatim from paper)

We adopt a comprehensive metric suite to capture different aspects of system performance. Throughput, measured in tokens per second, quantifies the raw processing capability of each framework and directly reflects the efficiency of GPU utilization and batch processing strategies. This metric is particularly important for understanding system capacity under high-load scenarios. Total time encompasses the end-to-end latency from request submission to complete response generation, capturing all system overheads including model loading, attention computation, KV Cache management, and result post-processing. For user-facing applications, we measure average latency decomposed into TTFT and inter-token latency, providing critical insights into perceived responsiveness.

Citation

@misc{he2025banaserve,
  title={BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2510.13223}
}

• arXiv: 2510.13223