Ffdl Platform Overhead

ffdl-platform-overhead

FfDL : A Flexible Multi-tenant Deep Learning Platform — Jayaram et al. (2019) (arXiv:1909.06526, 2019)

What this evaluates

Evaluates the runtime performance overhead and scalability of a containerized deep learning platform (FfDL) compared to bare-metal and specialized hardware, using standard image classification benchmarks. It measures how much throughput degrades when running DL training jobs in a Kubernetes-based multi-tenant environment versus direct execution.

Datasets

• ImageNet — total 1300000; splits: train (-1)

Metrics

• performance_overhead_pct (primary) — range: percent
  • Percentage decrease in training throughput relative to a bare-metal baseline. Calculated as ((baseline_throughput - platform_throughput) / baseline_throughput) * 100.
• throughput_images_per_sec — range: other
  • Number of images processed per second during model training. Measured directly from the training loop iterations and batch size.

Input / output format

Input: DL training job configuration specifying model architecture (VGG-16, Inception V3, ResNet-50), framework (Caffe v1.0, TensorFlow v1.5), GPU type (K80, P100, V100), number of learners, GPUs per learner, and batch size.

Output: Throughput in images processed per second, and percentage decrease in performance relative to a bare-metal baseline.

Scoring recipe

baseline = run_benchmark_baremetal(model, framework, gpu_config)
platform = run_benchmark_ffdl(model, framework, gpu_config)
overhead = ((baseline - platform) / baseline) * 100
return overhead

Common pitfalls

• Overhead varies significantly by GPU type and distributed configuration (learners/GPUs per learner), so reporting a single aggregate number masks significant variance.
• Baseline uses manually managed bare-metal servers rather than a standardized containerized setup, potentially attributing K8s/Docker overhead incorrectly.
• Throughput is highly sensitive to CPU thread allocation and batch size; failing to saturate the GPU will artificially inflate measured overhead.

Evidence (verbatim from paper)

Performance is quantified as images processed/sec for training. Caffe v1.0 and TensorFlow v1.5 were used. ... From Table 1, we observe that performance overhead induced by FfDL is minimal (up to ≈ 5%).

Citation

@misc{jayaram2019ffdl,
  title={FfDL : A Flexible Multi-tenant Deep Learning Platform},
  author={Jayaram et al. (2019)},
  year={2019},
  note={arXiv:1909.06526}
}

• arXiv: 1909.06526