Compression Benchmark Eval

compression-benchmark-eval

AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles — Liu et al. (2020) (arXiv:2006.04432, 2020)

What this evaluates

Evaluates the trade-offs between model accuracy and resource efficiency when applying various DNN compression techniques on mobile hardware. It measures how different compression methods affect inference speed, energy consumption, and storage footprint across standard vision and audio datasets.

Datasets

• CIFAR-10 — total ?; splits: test (-1)
• MNIST — total ?; splits: test (-1)
• CIFAR-100 — total ?; splits: test (-1)
• ImageNet — total ?; splits: test (-1)
• UbiSound — total ?; splits: test (-1)
• Har — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified samples out of the total test set. Calculated as (correct_predictions / total_samples) * 100%.
• latency — range: other
  • Average time required to process a single inference request on the target mobile hardware, measured in milliseconds.
• energy_cost — range: other
  • Total electrical energy consumed by the device during the inference process, measured in millijoules.
• storage — range: other
  • Size of the compressed model parameters stored on the device, measured in megabytes. Feature storage is typically negligible and ignored.

Input / output format

Input: Compressed DNN model (e.g., AlexNet, ResNet) compiled into an Android project, receiving raw input samples (images or audio) from the evaluation dataset.

Output: Class predictions for accuracy calculation, plus empirical runtime measurements: parameter storage size (MB), average inference latency per sample (ms), and total energy consumed during inference (mJ).

Scoring recipe

1. Deploy compressed DNN on target mobile platform.
2. Run inference on the full test dataset 10 times to account for device workload variance.
3. Compute accuracy = (correct_predictions / total_samples) * 100.
4. Record average latency (ms) and total energy consumption (mJ) across the 10 runs.
5. Measure model file size on disk for storage (MB).
6. Return [accuracy, storage_mb, latency_ms, energy_mj].

Common pitfalls

• Energy and latency measurements are highly sensitive to background OS processes and thermal throttling; the protocol requires averaging over 10 runs to mitigate workload variance.
• Storage is technically split into parameter storage (S_p) and feature storage (S_f), but S_f is typically in the KB range and ignored in later experiments, so only S_p (MB) is reported.
• Baseline compression techniques use empirically tuned, layer-specific hyperparameters rather than jointly optimized ones, which may underrepresent their true potential compared to the automated framework.

Evidence (verbatim from paper)

In our experiment, we study the performance differences of the state-of-the-art DNN compression techniques in terms of user demand metrics, i.e., accuracy A, storage S, latency T, and energy cost E. For this benchmark, we use the default compression hyperparameters (e.g., k in both W1f and W2) for a fair comparison.

Citation

@misc{liu2020adadeep,
  title={AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles},
  author={Liu et al. (2020)},
  year={2020},
  note={arXiv:2006.04432}
}

• arXiv: 2006.04432