Docker Dl Performance Eval

docker-dl-performance-eval

Performance Evaluation of Deep Learning Tools in Docker Containers — Xu et al. (2017) (arXiv:1711.03386, 2017)

What this evaluates

Evaluates the performance overhead of Docker containers on deep learning workloads by benchmarking CPU, GPU, I/O, and training speed of representative neural networks (FCN, CNN, RNN) across different frameworks.

Datasets

• MNIST — total 60000; splits: train (-1)
• Cifar10 — total ?; splits: train (-1)
• PTB — total ?; splits: train (-1)

Metrics

• second per batch (primary) — range: seconds
  • The wall-clock time required to process a single training batch through the network. Lower values indicate faster training.
• GFlops — range: GFlops
  • Giga floating point operations per second, used for CPU/GPU compute benchmarks (HPL, HPCG, matrix multiplication).
• I/O latency — range: milliseconds
  • Time taken for disk read/write operations, measured via dd and ioping tools.

Input / output format

Input: Neural network architecture specifications (layer dimensions, activation functions), dataset samples (images or character sequences), and batch size.

Output: Training time per batch in seconds.

Scoring recipe

def compute_metric(predictions, gold):
    # This benchmark measures wall-clock training time, not prediction accuracy.
    # The metric is directly recorded as the time taken to process one batch.
    return measured_time_per_batch

Common pitfalls

• Overhead is workload-dependent: compute-intensive tasks show negligible overhead, while I/O tasks may show different behavior due to caching.
• Standard Docker cannot access GPUs; the experiments specifically use NVIDIA Docker, which acts as a thin wrapper to load GPU drivers.
• Results are averaged over 20 runs, not reported as single-run values.

Evidence (verbatim from paper)

We measure the speed in unit of second per batch. All reported results are the average of 20 runs unless otherwise specified.

Citation

@misc{xu2017dockerdlperf,
  title={Performance Evaluation of Deep Learning Tools in Docker Containers},
  author={Xu et al. (2017)},
  year={2017},
  note={arXiv:1711.03386}
}

• arXiv: 1711.03386