Speed Bench Eval

speed-bench-eval

SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding — Abramovich et al. (2026) (arXiv:2604.09557, 2026)

What this evaluates

Evaluates the accuracy and throughput of speculative decoding methods across diverse semantic domains and varying input sequence lengths. It probes how draft length, batch size, vocabulary pruning, and inference frameworks impact real-world serving efficiency compared to baseline autoregressive generation.

Datasets

• SPEED-Bench — total ?; splits: Qualitative Split (-1), Throughput Split (-1)

Metrics

• AL (primary) — range: other (0 to DL)
  • The mean number of tokens accepted by the target model per draft step across a batch or domain. Calculated by averaging the count of verified tokens per generation instance.
• Speedup — range: other (multiplier)
  • Ratio of baseline autoregressive generation time to speculative decoding generation time. Reported as a multiplier where values >1 indicate faster inference.

Input / output format

Input: Text prompts from diverse semantic domains (e.g., Coding, Math, Multilingual, RAG) with varying input sequence lengths (ISL buckets like 2k, 8k).

Output: Generated token sequences using speculative decoding (draft chains) with configurable draft length (DL) and batch size (BS).

Scoring recipe

def compute_metrics(predictions, gold, baseline_times, sd_times):
    accepted_tokens = [len(p) for p in predictions]
    al = sum(accepted_tokens) / len(predictions)
    speedup = sum(baseline_times) / sum(sd_times)
    return {"AL": al, "Speedup": speedup}

Common pitfalls

• Using synthetic/random token inputs instead of the provided semantic splits overestimates throughput by ~23% due to skewed acceptance rates and unrealistic expert routing in MoE models.
• Evaluating only at batch size 1 or low draft lengths misses batch-size-dependent optimal DL shifts and concurrency bottlenecks present in production workloads.
• Aggressive vocabulary pruning (e.g., to 32k tokens) severely degrades performance on low-diversity or multilingual domains, which standard benchmarks often miss.

Evidence (verbatim from paper)

We evaluate speculation accuracy and system speedups across the Qualitative Split. All measurements use a batch size of 32 to simulate realistic workloads, utilizing TensorRT-LLM and SGLang for Qwen3 models due to engine constraints. Table 1 presents the average ALs and speedups using a DL of 3.

Citation

@misc{abramovich2026speedbench,
  title={SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding},
  author={Abramovich et al. (2026)},
  year={2026},
  note={arXiv:2604.09557}
}

• arXiv: 2604.09557