Dl Framework Benchmark Eval

dl-framework-benchmark-eval

A detailed comparative study of open source deep learning frameworks — Al-Bdour et al. (2019) (arXiv:1903.00102, 2019)

What this evaluates

Evaluates the execution speed and hardware resource utilization of three open-source deep learning frameworks (TensorFlow, Theano, CNTK) across standard computer vision, NLP, and custom datasets.

Datasets

• MNIST — total ?; splits: train (-1), test (-1)
• CIFAR-10 — total ?; splits: train (-1), test (-1)
• IMDB — total ?; splits: train (-1), test (-1)
• Self-Driving Car — total ?; splits: train (-1)
• Penn TreeBank — total ?; splits: train (-1), test (-1)

Metrics

• processing_time (primary) — range: seconds
  • Wall-clock time in seconds (or hours) required to complete the training process for a given model on a specified hardware environment.
• accuracy — range: percent
  • Percentage of correctly classified samples on the test set.
• perplexity — range: other
  • Exponential of the average negative log-likelihood of the test set, used for language modeling evaluation.
• cpu_utilization — range: percent
  • Percentage of CPU capacity actively used during training.
• gpu_utilization — range: percent
  • Percentage of GPU compute capacity actively used during training.
• memory_utilization — range: percent
  • Percentage of system or GPU memory consumed during training.

Input / output format

Input: Dataset, framework (CNTK, TensorFlow, Theano), hardware environment (CPU/GPU), and thread count (for CPU runs).

Output: Processing time in seconds/hours, CPU/GPU/memory utilization percentages, final model accuracy or perplexity, and number of epochs to convergence.

Scoring recipe

def evaluate_framework(dataset, framework, hardware, threads):
    start_time = time.time()
    model = build_model(framework, dataset)
    train(model, dataset, hardware, threads)
    elapsed = time.time() - start_time
    util_cpu = measure_cpu_usage()
    util_gpu = measure_gpu_usage()
    util_mem = measure_memory_usage()
    acc = compute_accuracy(model, dataset.test)
    return elapsed, util_cpu, util_gpu, util_mem, acc

Common pitfalls

• Hardware specifications (CPU cores, GPU model, RAM) drastically change results; comparisons are only valid on identical hardware.
• CNTK's Python API does not support CPU multithreading, so CPU results for CNTK are limited to single-thread or default core counts.
• The 'Self-Driving Car' dataset is a custom, non-standard dataset, making cross-study comparisons difficult.
• Processing time includes training time, not inference time, which may not reflect deployment latency.

Evidence (verbatim from paper)

Table 4 shows the CPU and GPU processing times for each dataset. ... The metrics measurement of each framework was conducted to explain the failure of one of the selected frameworks.

Citation

@misc{albdour2019frameworks,
  title={A detailed comparative study of open source deep learning frameworks},
  author={Al-Bdour et al. (2019)},
  year={2019},
  note={arXiv:1903.00102}
}

• arXiv: 1903.00102