Flying Serving Eval

flying-serving-eval

FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving — Gao et al. (2026) (arXiv:2602.22593, 2026)

What this evaluates

Evaluates the runtime performance of an LLM serving engine under bursty, heterogeneous, and long-context workloads. It probes the system's ability to dynamically switch between data and tensor parallelism to optimize latency and throughput while maintaining memory efficiency compared to static and alternative dynamic baselines.

Datasets

• ShareGPT — total ?; splits: test (-1)
• CodeActInstruct — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)
• Synthetic Workloads — total ?; splits: test (-1)

Metrics

• TTFT (primary) — range: other
  • Latency from when a request arrives at the serving system to when the first output token is generated, including both queuing and prefill time.
• TPOT — range: other
  • Per-request average time-between-tokens during decoding, measured over consecutive output tokens after the first (i.e., inter-token interval).
• Peak generation throughput — range: other
  • Maximum aggregate output token rate (tokens/s) sustained by the system under load.
• Queue time — range: other
  • Time from request admission to first scheduling, isolating scheduler delay from execution time.

Input / output format

Input: Text prompts sampled from ShareGPT, CodeActInstruct, HumanEval, or synthetically generated with input lengths uniformly spanning [128, 4000] tokens and output lengths [64, 512] tokens, arriving in traces with alternating low (2–5 req/s) and high (10–30 req/s) load rates.

Output: Generated token sequences with precise per-token timestamps logged to compute inter-token intervals and generation rates.

Scoring recipe

def compute_metrics(requests):
    ttfts = [req.first_token_time - req.arrival_time for req in requests]
    tpots = [mean(req.token_timestamps[i] - req.token_timestamps[i-1]) for req in requests]
    queue_times = [req.scheduling_time - req.arrival_time for req in requests]
    total_tokens = sum(len(req.output) for req in requests)
    total_time = max(req.end_time) - min(req.arrival_time)
    throughput = total_tokens / total_time
    return {'TTFT': mean(ttfts), 'TPOT': mean(tpots), 'Queue Time': mean(queue_times), 'Throughput': throughput}

Common pitfalls

• Confusing TPOT (strict inter-token interval during decoding) with ILT (Inter-Token Latency), which aggregates compute, queueing, and batching effects and varies with scheduler decisions.
• Assuming static baselines are directly comparable without accounting for their fixed parallelism constraints, which rigidly limit maximum context length and force costly cold-start restarts when limits are exceeded.
• Overlooking that synthetic workloads are explicitly used to control arrival-time traces, as public datasets only provide request contents without realistic timing patterns.

Evidence (verbatim from paper)

We use standard streaming-inference metrics that quantify initial responsiveness and steady-state token generation: (i) Time To First Token (TTFT): latency from when a request arrives at the serving system to when the first output token is generated (including both queuing and prefill). (ii) Time Per Output Token (TPOT): the per-request average time-between-tokens during decoding, measured over consecutive output tokens after the first (i.e., inter-token interval). (iii) Peak generation throughput: maximum aggregate output token rate (tokens/s) sustained by the system under load.

Citation

@misc{gao2026flyingserving,
  title={FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2602.22593}
}

• arXiv: 2602.22593