Hies Pruning Eval

hies-pruning-eval

Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning — Choi et al. (2025) (arXiv:2510.13832, 2025)

What this evaluates

Evaluates the accuracy and stability of transformer head pruning methods (specifically HIES vs. baselines) across NLP, vision, and multimodal benchmarks at fixed sparsity ratios (10%, 30%, 50%).

Datasets

• GLUE (SST-2, CoLA, MRPC, QQP, STS-B, QNLI, MNLI, RTE) — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• Winogrande — total ?; splits: test (-1)
• ARC-e / ARC-c — total ?; splits: test (-1)
• OBQA — total ?; splits: test (-1)
• ImageNet1k — total ?; splits: test (-1)
• CIFAR-100 — total ?; splits: test (-1)
• Food-101 — total ?; splits: test (-1)
• Fashion MNIST — total ?; splits: test (-1)
• VizWiz-VQA — total ?; splits: test (-1)
• MM-Vet — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1] | percent
  • Task-specific metric: Accuracy for most tasks, Matthews correlation coefficient for CoLA, F1 score for MRPC, and Pearson correlation for STS-B. Averaged across tasks for the 'Average' column.
• Stability — range: percent
  • Percentage of the pruned model's task metric relative to the unpruned baseline's metric, indicating robustness to aggressive compression.

Input / output format

Input: Pruned transformer model (BERT_base, LLaMA-2_7B, ViT_Large, LLaVA-1.5_7B) fed with benchmark inputs (text, images, or image-text pairs).

Output: Task predictions (class labels, scores, or generated text) used to compute benchmark metrics.

Scoring recipe

def compute_accuracy(preds, gold, task):
    if task == 'CoLA': return matthews_corrcoef(gold, preds)
    if task == 'MRPC': return f1_score(gold, preds)
    if task == 'STS-B': return pearsonr(gold, preds)
    return accuracy_score(gold, preds)

def compute_stability(pruned_acc, base_acc):
    return (pruned_acc / base_acc) * 100

Common pitfalls

• Stability is a relative metric (pruned vs. unpruned accuracy), not absolute performance.
• GLUE tasks use different evaluation metrics (Matthews corr, F1, Pearson corr, Accuracy) that are averaged in the 'Average' column.
• Pruning ratios are fixed at 10/30/50% for BERT but extended to 60% for LLaMA-2.

Evidence (verbatim from paper)

Datasets. We evaluate on various widely-adopted benchmarks: GLUE*[Wang et al., [2018]], HellaSwag[Zellers et al., [2019]], Winogrande[Sakaguchi et al., [2020]], the AI2 Reasoning Challenge—ARC-e/ARC-c[Clark et al., [2018]], OBQA[Mihaylov et al., [2018]], ImageNet1k[Deng et al., [2009]], CIFAR-100[Krizhevsky, [2009]], Food-101[Bossard et al., [2014]], Fashion MNIST[Xiao et al., [2017]], VizWiz-VQA[Gurari et al., [2018]], and MM-Vet[Yu et al., [2024]]*. We evaluate HIES using two key metrics: model quality and stability.

Citation

@misc{choi2025hies,
  title={Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning},
  author={Choi et al. (2025)},
  year={2025},
  note={arXiv:2510.13832}
}

• arXiv: 2510.13832