qhjqhj00 mobile-dl-training-performance
Performance Analysis and Characterization of Training Deep Learning Models on Mobile Devices — Liu et al. (2019) (arXiv:1906.04278, 2019)
What this evaluates
This evaluation probes the hardware efficiency and resource constraints of training deep learning models on mobile SoCs. It measures how different model architectures and batch sizes impact GPU/CPU utilization, power/energy draw, and memory footprint during training.
Datasets
- Cifar-10 — total ?; splits: train (-1)
- Shakespeare — total ?; splits: train (-1)
- Atari 2600 games — total ?; splits: train (-1)
- SQuAD — total ?; splits: train (-1)
- IWSLT15 — total ?; splits: train (-1)
Metrics
GPU utilization(primary) — range: percent- Percentage of time the GPU is actively executing operations during a training step, measured via hardware counters.
CPU utilization— range: percent- Percentage of time each individual CPU core (Denver2 or A57) is active during training, measured via hardware counters.
Power consumption— range: Watts- Instantaneous power draw in Watts for CPU, GPU, and memory components, sampled at 5ms intervals.
Energy consumption— range: Joules- Total energy used per training step, calculated by summing power readings multiplied by the 5ms sampling interval.
Peak memory consumption— range: GB- Maximum RAM usage (GB) recorded during training, including parameters, gradients, input data, and intermediate feature maps.
Input / output format
Input: Deep learning model architectures (e.g., CNNs, RNNs, GANs) configured with specific datasets and batch sizes (1–64), executed on an NVIDIA TX2 SoC using TensorFlow v1.13.
Output: Hardware utilization percentages, power (W) and energy (J) consumption values, peak memory usage (GB), and a binary success/failure flag for training completion.
Scoring recipe
def evaluate_training_performance(model, dataset, batch_size):
# Skip first 3 steps for runtime warmup
for step in range(3):
model.train_step(dataset, batch_size)
metrics = {'gpu_util': [], 'cpu_util': [], 'power': [], 'energy': [], 'mem_peak': 0}
for step in range(3, total_steps):
gpu_u, cpu_u, power_w = read_hardware_counters()
metrics['gpu_util'].append(gpu_u)
metrics['cpu_util'].append(cpu_u)
metrics['power'].append(power_w)
metrics['energy'].append(power_w * 0.005)
metrics['mem_peak'] = max(metrics['mem_peak'], read_memory_usage())
try:
model.train_step(dataset, batch_size)
except MemoryError:
return {'success': False, 'metrics': metrics}
return {
'success': True,
'gpu_util_avg': mean(metrics['gpu_util']),
'cpu_util_avg': mean(metrics['cpu_util']),
'power_avg': mean(metrics['power']),
'energy_total': sum(metrics['energy']),
'mem_peak_gb': metrics['mem_peak'] / 1024
}
Common pitfalls
- Must skip the first 3 training steps as they are used by the TensorFlow runtime for hardware exploration and are not representative of steady-state performance.
- Large models (e.g., VGG19, BERT, ResNet152) will crash due to the 8GB memory limit; batch size must be carefully tuned to avoid OOM errors.
- CPU utilization is heterogeneous (Denver2 vs A57 cores) and does not scale linearly with batch size like GPU utilization does.
Evidence (verbatim from paper)
We study the training performance from the following perspectives: hardware (CPU and GPU) utilization, power consumption, and peak memory consumption.
Citation
@misc{liu2019performance,
title={Performance Analysis and Characterization of Training Deep Learning Models on Mobile Devices},
author={Liu et al. (2019)},
year={2019},
note={arXiv:1906.04278}
}
- arXiv: 1906.04278