Ralp Cnn Training Eval

ralp-cnn-training-eval

Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement — Park et al. (2019) (arXiv:1901.05803, 2019)

What this evaluates

Evaluates the training throughput and network communication efficiency of distributed CNN training frameworks under varying GPU counts and dataset complexities. Probes how well a system mitigates parameter server bottlenecks and scales across multiple concurrent workloads.

Datasets

• ImageNet-1K — total 1281166; splits: train (-1)
• ImageNet-22K — total ?; splits: train (-1)

Metrics

• throughput (images/sec) (primary) — range: other
  • Average number of images processed per second across all workers over 100 training iterations after a warmup period.
• transfer volume (GB/step) — range: other
  • Total data volume transferred between workers and parameter servers per training step.
• speedup — range: other
  • Ratio of RALP throughput to baseline (Distributed TensorFlow) throughput.

Input / output format

Input: CNN model architecture (e.g., VGG11, ResNet-50) and mini-batches of images from ImageNet-1K or ImageNet-22K, distributed across N GPUs in a parameter-server or all-reduce architecture.

Output: System-level performance metrics: training throughput (images/sec), inter-node communication volume (GB/step), and relative speedup compared to baselines.

Scoring recipe

def compute_throughput(model, dataloader, num_workers, batch_size):
    warmup_steps = 5
    measure_steps = 100
    total_images = 0
    for step in range(warmup_steps + measure_steps):
        batch = next(dataloader)
        train_step(model, batch)
        if step >= warmup_steps:
            total_images += batch_size
    return total_images / measure_steps

def compute_speedup(ralp_throughput, baseline_throughput):
    return ralp_throughput / baseline_throughput

Common pitfalls

• The paper explicitly states that model accuracy/convergence is not reported because execution changes do not affect accuracy; evaluators should not expect or request accuracy metrics.
• RALP has two distinct GPU allocation configurations (RALP-H: N/2 workers + 1 PS; RALP-N: N-1 workers + 1 PS) that must be clearly distinguished when comparing against Horovod or the baseline.
• Throughput is averaged over exactly 100 iterations after a warmup phase; skipping warmup or averaging over different iteration counts will skew results.

Evidence (verbatim from paper)

As performance metric, we use throughput (images processed per second) across all workers at the training iteration. At the same time, we measure the data volume transferred between workers and PSes to show the effect RALP has on mitigating network traffic. We collect these metrics averaged over 100 iterations after warming up the system through the first few iterations. We do not report convergence or accuracy of the trained model as there is no change in execution and thus, the accuracy remains the same.

Citation

@misc{park2019ralp,
  title={Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement},
  author={Park et al. (2019)},
  year={2019},
  note={arXiv:1901.05803}
}

• arXiv: 1901.05803