Zero Shot Adjustable Acceleration Eval

zero-shot-adjustable-acceleration-eval

Efficient Large Language Models with Zero-Shot Adjustable Acceleration — Sajjad Kachuee, Mohammad Sharifkhani (2025) (arXiv:2509.01190, 2025)

What this evaluates

This evaluation protocol assesses the capability of large language models to maintain task performance while dynamically pruning hidden activations during inference. It probes the model's robustness across natural language understanding, text generation, and instruction-tuning tasks under varying computational constraints and acceleration ratios.

Datasets

• IMDB — total ?; splits: train (-1), test (-1)
• GLUE — total ?; splits: validation (-1); repo https://gluebenchmark.com/
• WikiText-103 — total ?; splits: test (-1)
• Penn Treebank (PTB) — total ?; splits: test (-1)
• One Billion Word (1BW) — total ?; splits: test (-1)
• LAMBADA — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels out of total instances.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall, computed per task and averaged.
• Perplexity (PPL) — range: [0, inf)
  • Exponential of the average negative log-likelihood of the ground truth tokens: exp(-1/N * sum(log p(x_i))).

Input / output format

Input: Raw text inputs including prompts, questions, or sentence pairs depending on the task (classification, language modeling, or instruction-following).

Output: Predicted class labels for classification tasks, token probabilities or next-token predictions for language modeling, and selected multiple-choice options for MMLU.

Scoring recipe

def compute_metrics(predictions, gold_labels, task_type):
    if task_type in ['classification', 'MMLU']:
        return sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    elif task_type in ['MRPC', 'QQP']:
        tp = sum(p == g for p, g in zip(predictions, gold_labels))
        prec = tp / max(1, sum(predictions))
        rec = tp / max(1, sum(gold_labels))
        return 2 * prec * rec / max(1e-9, prec + rec)
    elif task_type in ['language_modeling']:
        log_probs = [math.log(p) for p in predictions]
        return math.exp(-sum(log_probs) / len(log_probs))

Common pitfalls

• GLUE results are reported on validation sets rather than the official test set due to server access restrictions.
• The preservation rate hyperparameter ($\alpha$) requires dataset-specific tuning because sequence lengths vary significantly across benchmarks.
• 4-bit quantized models are more sensitive to structural variations from activation pruning than full-precision models, requiring careful $\alpha$ bounds.

Evidence (verbatim from paper)

Performance is reported as F1-score for MRPC and QQP, and accuracy for all other tasks.

Citation

@misc{kachuee2025efficient,
  title={Efficient Large Language Models with Zero-Shot Adjustable Acceleration},
  author={Sajjad Kachuee, Mohammad Sharifkhani (2025)},
  year={2025},
  note={arXiv:2509.01190}
}

• arXiv: 2509.01190