Plm Structural Pruning Eval

plm-structural-pruning-eval

Structural Pruning of Pre-trained Language Models via Neural Architecture Search — Klein et al. (2024) (arXiv:2405.02267, 2024)

What this evaluates

Evaluates structural pruning methods for pre-trained language models (BERT-base, RoBERTa-base) across eight text classification tasks. It measures the trade-off between model size (parameter count) and task performance (validation error) to identify Pareto-optimal sub-networks.

Datasets

• eight text classification tasks — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Hypervolume (primary) — range: [0, 4]
  • HV(P_f|r) = λ(∪_{y∈P_f}[y, r]), where λ is the M-th dimensional Lebesgue measure. Objectives are first normalized via Quantile normalization to [0,1].

Input / output format

Input: Text classification instances (input text and label) from eight tasks.

Output: Pruned sub-network architectures (binary masks for attention heads, feed-forward units, or layers) with fine-tuned weights, evaluated on validation/test sets.

Scoring recipe

def compute_hypervolume(pareto_set, reference_point=(2,2)):
    # 1. Quantile-normalize parameter count and error across all methods/repetitions to [0,1]
    # 2. Define reference point r = (2, 2)
    # 3. Compute Lebesgue measure of union of hyper-rectangles [y, r] for y in pareto_set
    # 4. Return HV value
    pass

Common pitfalls

• Hypervolume must be computed on objectives normalized via Quantile normalization, not raw values.
• The reference point is fixed at (2, 2) after normalization, yielding a maximum possible HV of 4, not the standard (1, 1).
• Results are aggregated by averaging ranks across datasets and time steps, not just by final HV values.

Evidence (verbatim from paper)

To quantify the performance of a Pareto set $P_{f}$ , we compute for each Pareto set the Hypervolume (Zitzler et al., 2003) and report the regret, i.e. the difference to the best possible Hypervolume averaged across all repetitions. Given a reference point $\mathbf{r} \in \mathbb{R}^{M}$ , the Hypervolume $HV(P_{f}|\mathbf{r}) = \lambda (\cup_{\mathbf{y} \in P_{f}}[\mathbf{y},\mathbf{r}])$ is defined as the $M$ -th dimensional Lebesgue measure $\lambda$ between the Pareto set $P_{f}$ and the reference point $\mathbf{r}$ (see Figure 4 for an example). To compute the Hypervolume, we first normalize each objective based on all observed values across all methods and repetitions via Quantile normalization. This results in a uniform distribution between $[0,1]$ , and we use $r = (2,2)$ as reference point, which means the highest possible Hypervolume would be 4.

Citation

@misc{klein2024structuralpruning,
  title={Structural Pruning of Pre-trained Language Models via Neural Architecture Search},
  author={Klein et al. (2024)},
  year={2024},
  note={arXiv:2405.02267}
}

• arXiv: 2405.02267