Deep Compression Eval

deep-compression-eval

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding — Song Han et al. (2015) (arXiv:1510.00149, 2015)

What this evaluates

Evaluates the effectiveness of a three-stage neural network compression pipeline (pruning, trained quantization, and Huffman coding) in reducing model storage size while preserving classification accuracy on standard computer vision benchmarks.

Datasets

• MNIST — total ?; splits: (unstated)
• ImageNet (ILSVRC-2012) — total 1250000; splits: train (1200000), val (50000)

Metrics

• Top-1 Accuracy (primary) — range: percent
  • Percentage of images where the predicted class matches the ground truth class. Calculated as (correct predictions / total images) * 100.
• Compress Rate — range: other
  • Ratio of the original model parameter storage size to the final compressed file size, including overhead from the weight codebook and sparse index arrays.

Input / output format

Input: Raw image pixels fed into a standard Caffe neural network architecture (LeNet-300-100, LeNet-5, AlexNet, or VGG-16).

Output: Predicted class label for each input image.

Scoring recipe

def compute_metrics(predictions, gold_labels, original_model, compressed_model_file):
    # Top-1 Accuracy
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    top1_acc = (correct / len(gold_labels)) * 100
    
    # Compress Rate
    original_size_bytes = sum(p.numel() for p in original_model.parameters()) * 4  # float32 baseline
    compressed_size_bytes = len(compressed_model_file)
    compress_rate = original_size_bytes / compressed_size_bytes
    
    return top1_acc, compress_rate

Common pitfalls

• Reporting compression as a percentage reduction instead of the multiplier (e.g., 35×) used in the paper.
• Failing to account for the storage overhead of the codebook and sparse index arrays when calculating the final compressed size.
• Assuming accuracy drops are acceptable without performing the specified fine-tuning step after pruning and quantization.

Evidence (verbatim from paper)

We use the AlexNet Caffe model as the reference model, which has 61 million parameters and achieved a top-1 accuracy of 57.2% and a top-5 accuracy of 80.3%. The network parameters and accuracy before and after pruning are shown in Table 1. The compression pipeline saves network storage by 35× to 49× across different networks without loss of accuracy.

Citation

@misc{han2015deepcompression,
  title={Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding},
  author={Song Han et al. (2015)},
  year={2015},
  note={arXiv:1510.00149}
}

• arXiv: 1510.00149