Consensus Layer Pruning Eval

consensus-layer-pruning-eval

Layer Pruning with Consensus: A Triple-Win Solution — Giusti Mugnaini et al. (2024) (arXiv:2411.14345, 2024)

What this evaluates

Evaluates a multi-metric layer pruning method (Consensus) on image classification models, measuring trade-offs between computational efficiency (FLOPs reduction) and predictive performance (accuracy drop), while also assessing robustness against adversarial and out-of-distribution attacks.

Datasets

• CIFAR-10 — total ?; splits: test (-1)
• ImageNet — total ?; splits: val (-1)
• CIFAR-10.2 — total ?; splits: test (-1)
• CIFAR-C — total ?; splits: test (-1)
• ImageNet-C — total ?; splits: test (-1)

Metrics

• Δ Acc. (difference in accuracy) (primary) — range: percent
  • Percentage point difference between pruned model accuracy and unpruned baseline accuracy: (Acc_pruned - Acc_original) * 100. Positive values indicate improvement, negative indicate drop.
• FLOPs reduction — range: percent
  • Percentage reduction in floating-point operations relative to the original unpruned model: ((FLOPs_original - FLOPs_pruned) / FLOPs_original) * 100.
• Mean accuracy across benchmarks — range: percent
  • Average classification accuracy computed across all specified adversarial/OOD benchmarks (CIFAR-10.2, CIFAR-C, ImageNet-C, FGSM) for a given pruning criterion.

Input / output format

Input: Pruned ResNet architectures (ResNet32, ResNet44, ResNet50, ResNet56) evaluated on clean and adversarial/out-of-distribution image datasets.

Output: Tables reporting Δ Acc. (in percentage points) and FLOPs reduction (%) at matched compression levels, plus mean accuracy scores across robustness benchmarks.

Scoring recipe

def compute_delta_acc(original_acc, pruned_acc):
    return (pruned_acc - original_acc) * 100

def compute_flops_reduction(orig_flops, pruned_flops):
    return ((orig_flops - pruned_flops) / orig_flops) * 100

def compute_robustness_acc(model, benchmarks):
    accs = [model.evaluate(b) for b in benchmarks]
    return sum(accs) / len(accs)

Common pitfalls

• Reporting absolute accuracy instead of the required delta accuracy (Δ Acc.) relative to the unpruned baseline.
• Comparing pruning methods at mismatched FLOP reduction percentages without exact matching or interpolation.
• Using incorrect severity levels for CIFAR-C (severity 4) and ImageNet-C (severity 5) when computing robustness metrics.

Evidence (verbatim from paper)

To assess the predictive ability of the unpruned models against their pruned versions, we adhere to standard practices by reporting the difference in accuracy [20, 42]. For the semantic-preserving attacks CIFAR-C and ImageNet-C, we report the average across all possible attacks as suggested by previous work [21]. Regardless of the dataset, negative values mean a decrease in accuracy, while positive values denote an improvement, both measured in percentage points (pp).

Citation

@misc{giustimugnaini2024layerpruning,
  title={Layer Pruning with Consensus: A Triple-Win Solution},
  author={Giusti Mugnaini et al. (2024)},
  year={2024},
  note={arXiv:2411.14345}
}

• arXiv: 2411.14345