Smtm Mobile Cnn Eval

smtm-mobile-cnn-eval

Boosting Mobile CNN Inference through Semantic Memory — Li et al. (2021) (arXiv:2112.02644, 2021)

What this evaluates

Evaluates the latency reduction, accuracy loss, memory overhead, energy saving, and early exit performance of a semantic memory caching mechanism (SMTM) for accelerating CNN inference on mobile devices.

Datasets

• UCF101 — total 70928; splits: test (70928)
• CIFAR-100 (long-tail) — total 1442; splits: test (1442)

Metrics

• latency reduction (primary) — range: percent
  • Percentage reduction in average processing time per frame compared to the no-cache baseline. Calculated as (1 - T_SMTM / T_baseline) * 100%.
• accuracy loss — range: percent
  • Drop in top-1 classification accuracy compared to the baseline model without caching.
• memory overhead — range: other
  • Additional RAM usage in megabytes required to store semantic centers and frequency tables.
• energy saving — range: percent
  • Percentage reduction in energy consumption measured via on-device PMIC (voltage × current × time).
• early exit ratio — range: percent
  • Percentage of input frames that trigger an early exit at a specific network layer based on semantic similarity threshold τ.

Input / output format

Input: Raw video frames (224×224 for UCF101, 32×32 for CIFAR-100) processed by CNN architectures (AlexNet, GoogleNet, ResNet50, MobileNet V2, VGG16).

Output: Top-1 class prediction and early-exit flag indicating whether inference terminates at the current layer based on semantic similarity threshold τ.

Scoring recipe

# Compute baseline metrics (no-cache CPU/GPU)
T_base = mean(latency(no_cache_baseline))
E_base = mean(energy(no_cache_baseline))
Acc_base = top1_accuracy(no_cache_baseline)

# Compute SMTM metrics
T_smtm = mean(latency(smtm_predictions))
E_smtm = mean(energy(smtm_predictions))
Acc_smtm = top1_accuracy(smtm_predictions)
Exit_counts = count_exits_per_layer(smtm_predictions)

latency_reduction = (1 - T_smtm / T_base) * 100
accuracy_loss = Acc_base - Acc_smtm
energy_saving = (1 - E_smtm / E_base) * 100
early_exit_ratio = Exit_counts / total_samples * 100

Common pitfalls

• Fair comparison with DeepCache/DeepMon requires matching ncnn configurations and explicitly disabling SIMD acceleration, as stated in the experimental setup.
• Energy measurement cannot rely on single inferences due to PMIC sampling limits; the device must be forced into an infinite inference loop to average voltage and current.
• The CIFAR-100 evaluation uses a custom long-tail split of 1,442 images rather than the standard 10,000-image test set.

Evidence (verbatim from paper)

We use five metrics to comprehensively evaluate the performance of SMTM: latency reduction (Section 8.2), accuracy loss (Section 8.3), memory overhead (Section 8.4), energy saving (Section 8.5), and early exit ratio (Section 8.6).

Citation

@misc{li2021boosting,
  title={Boosting Mobile CNN Inference through Semantic Memory},
  author={Li et al. (2021)},
  year={2021},
  note={arXiv:2112.02644}
}

• arXiv: 2112.02644