Adadecode Eval

adadecode-eval

AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism — Wei et al. (2025) (arXiv:2506.03700, 2025)

What this evaluates

Evaluates the inference speedup and output consistency of adaptive layer parallelism for LLM decoding compared to standard autoregressive generation.

Datasets

• HumanEval — total ?; splits: test (-1)

Metrics

• speedup (primary) — range: other
  • Ratio of inference time of standard autoregressive decoding to AdaDecode inference time.
• output consistency ratio — range: percent
  • Percentage of generated outputs that exactly match the output of standard autoregressive decoding.
• early prediction rate — range: percent
  • Percentage of decoding steps where early prediction is triggered based on the confidence threshold γ.
• verification rejection rate — range: percent
  • Percentage of early-predicted tokens that fail verification and are rejected during decoding.

Input / output format

Input: Text prompts or code generation tasks fed to the LLM backbone (e.g., CodeLlama-34B-Instruct, Llama3.1-8B-Instruct).

Output: Autoregressive token sequences generated by the model.

Scoring recipe

speedup = time_vanilla_ar / time_adadecode
consistent = sum(1 for p, g in zip(predictions, gold) if p == g)
consistency_ratio = (consistent / len(predictions)) * 100
early_prediction_rate = (steps_with_early_pred / total_steps) * 100
verification_rejection_rate = (rejected_tokens / total_early_pred_tokens) * 100

Common pitfalls

• FP16 numerical precision can cause minor deviations from theoretical output parity, affecting consistency ratio measurements.
• Hardware FLOP bottlenecks can artificially lower measured speedup for methods requiring parallel n-gram generation.
• Hyperparameter γ significantly impacts early prediction rate and rejection rate; results are sensitive to its exact value.

Evidence (verbatim from paper)

As presented in[Table 2], our method consistently delivers superior speedup compared to all baseline approaches regardless of the backbone model size, achieving up to $1.73 imes$ speedup compared to standard autoregressive decoding. As shown in[Figure 4], we empirically evaluate the output consistency ratio of all baseline methods against the vanilla decoding and report their corresponding speedup ratios on the HumanEval benchmark with CodeLlama-34B-Instruct as the backbone.

Citation

@misc{wei2025adadecode,
  title={AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2506.03700}
}

• arXiv: 2506.03700