Hiespec Throughput Eval

hiespec-throughput-eval

HiSpec: Hierarchical Speculative Decoding for LLMs — Kumar et al. (2025) (arXiv:2510.01336, 2025)

What this evaluates

Evaluates the inference throughput speedup of hierarchical speculative decoding against vanilla auto-regressive decoding and other acceleration baselines. It probes the method's ability to accelerate token generation across dialogue, summarization, code generation, and mathematical reasoning tasks without relying on auxiliary draft models.

Datasets

• ShareGPT — total ?; splits: test (-1)
• CNN/DM — total ?; splits: test (-1)
• XSum — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)

Metrics

• Speedup (vs. Vanilla) (primary) — range: ratio
  • Ratio of the evaluated method's throughput to the throughput of vanilla auto-regressive decoding on the same hardware and configuration. Higher values indicate greater acceleration.

Input / output format

Input: Natural language prompts for dialogue continuation, text summarization, code completion, and mathematical reasoning problems.

Output: Autoregressively generated token sequences until an end-of-sequence token or maximum length is reached.

Scoring recipe

def compute_speedup(method_throughput, vanilla_throughput):
    return method_throughput / vanilla_throughput
# Evaluated per dataset and model size combination

Common pitfalls

• Speedup is a relative ratio to vanilla decoding, not absolute tokens/sec, making cross-hardware comparisons invalid.
• The table only reports throughput; accuracy preservation is claimed in text but not quantified in this specific evaluation table.
• Comparisons across different base model sizes (e.g., 7B vs 70B) are presented in separate blocks, which can obscure per-model scaling trends.

Evidence (verbatim from paper)

HiSpec improves throughput: Table[1] compares the performance of HiSpec against prior works which accelerate draft token generation. HiSpec improves throughput by $1.7 imes$ on average and up to $2.08 imes$ compared to vanilla auto-regressive decoding. Table 1: Throughput of HiSpec and other competing approaches relative to vanilla auto-regressive decoding (higher is better), across dialogue (ShareGPT), text summarization (CNN/DM, XSum), code generation (HumanEval), and mathematical reasoning (GSM8K) tasks. | Method | Speedup (vs. Vanilla) |

Citation

@misc{kumar2025hiespec,
  title={HiSpec: Hierarchical Speculative Decoding for LLMs},
  author={Kumar et al. (2025)},
  year={2025},
  note={arXiv:2510.01336}
}

• arXiv: 2510.01336