Total Latency

total-latency

Efficient LLM Inference over Heterogeneous Edge Networks with Speculative Decoding — Zhu et al. (2025) (arXiv:2510.11331, 2025)

What this evaluates

Measures the end-to-end serving latency of an LLM inference system deployed over heterogeneous edge networks using speculative decoding. It probes how well pipeline parallelism, adaptive batching, and wireless resource allocation reduce total time-to-output compared to sequential or fixed-strategy baselines.

Datasets

• Synthetic Edge Inference Tasks — total 100; splits: test (100)

Metrics

• total latency (primary) — range: other
  • Sum of communication latency (uplink transmission time for input sequences) and inference latency (time for draft generation and verification across SBS and MBS GPUs). Measured in seconds or milliseconds.

Input / output format

Input: Randomly generated text tasks with input length sampled uniformly from [1, I_max] and expected output length from [1, O_max].

Output: Generated text tokens up to the expected output length.

Scoring recipe

def compute_total_latency(tasks, system_config):
    comm_latency = sum(upload_time(task.input, bandwidth) for task in tasks)
    inference_latency = 0
    for batch in system_config.batches:
        draft_time = run_draft_model(batch, system_config.draft_model)
        verify_time = run_verify_model(batch, system_config.verify_model)
        inference_latency += max(draft_time, verify_time) # pipelined
    return comm_latency + inference_latency

Common pitfalls

• Latency encompasses both wireless communication and GPU inference time; isolating only compute time misrepresents the system bottleneck.
• Acceptance rate (α) is task-dependent and heavily influences latency; reporting results without specifying α or task difficulty is misleading.
• Synthetic input/output lengths are uniformly sampled, so latency scaling may not reflect real-world LLM query distributions.

Evidence (verbatim from paper)

Fig. [6] presents the impacts of the acceptance rate (i.e., α) on total latency for both the proposed serving mechanism and AD-based serving mechanism for three draft–verify model pairs. ... It is observed that compared to the AD-based serving mechanism, the proposed serving mechanism achieves consistently lower latency across all draft-verify model combinations and acceptance rates.

Citation

@misc{zhu2025efficient,
  title={Efficient LLM Inference over Heterogeneous Edge Networks with Speculative Decoding},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2510.11331}
}

• arXiv: 2510.11331