Mxnet Framework Benchmark Eval

mxnet-framework-benchmark-eval

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems — Chen et al. (2015) (arXiv:1512.01274, 2015)

What this evaluates

Evaluates the raw execution speed, memory footprint, and distributed scalability of the MXNet deep learning framework against Torch7, Caffe, and TensorFlow. It measures how efficiently the library handles standard convolutional neural network architectures and large-scale image classification tasks across single and multiple GPU nodes.

Datasets

• convnet-benchmarks — total ?; splits: (unstated)
• ILSVRC12 — total 1300000; splits: (unstated)

Metrics

• forward-backward performance (primary) — range: other
  • Wall-clock time in seconds to complete one forward and backward pass through the network architecture.
• internal memory usage — range: other
  • Peak internal memory consumption in megabytes during forward and forward-backward passes, explicitly excluding output variables.
• training time per epoch — range: other
  • Average wall-clock time in seconds to process one full pass through the dataset.

Input / output format

Input: Pre-configured neural network architectures (e.g., VGG, GoogLeNet) with fixed hyperparameters (learning rate 0.05, momentum 0.9, weight decay 1e-4) and batch sizes (32 for benchmarks, 36 per GPU for ILSVRC12) fed with image data.

Output: Execution time per forward-backward pass, peak internal memory usage in MB, and convergence curves showing training progress over data passes.

Scoring recipe

def evaluate_framework(model, data_loader, device, batch_size):
    start = time.time()
    for batch in data_loader:
        outputs = model(batch)
        loss = compute_loss(outputs, batch.labels)
        loss.backward()
    fb_time = (time.time() - start) / len(data_loader)
    mem_usage = get_internal_memory_usage(exclude_outputs=True)
    epoch_time = get_average_epoch_time()
    return {'forward_backward_time': fb_time, 'memory_mb': mem_usage, 'epoch_time': epoch_time}

Common pitfalls

• Comparisons are invalid if frameworks use different CUDA/CuDNN versions (e.g., TensorFlow was tested with older versions, artificially slowing it down).
• Memory usage figures explicitly exclude output variables, so they represent a lower bound of total VRAM consumption.
• Distributed training shows slower initial convergence than single-machine training, so evaluating only early epochs misrepresents scalability.

Evidence (verbatim from paper)

Figure 6: Compare MXNet to others on a single forward-backward performance. We fist compare MXNet with Torch7, Caffe, and TensorFlow on the popular “convnet-benchmarks”. All these systems are compiled with CUDA 7.5 and CUDNN 3 except for TensorFlow, which only supports CUDA 7.0 and CUDNN 2. We use batch size 32 for all networks and run the experiments on a single Nvidia GTX 980 card. Results are shown in Figure 7. As expected that MXNet has similar performance comparing to Torch7 and Caffe, because most computations are spent on the CUDA/CUDNN kernels. TensorFlow is always 2x slower, which might be due its use of a lower CUDNN version.

Citation

@misc{chen2015mxnet,
  title={MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems},
  author={Chen et al. (2015)},
  year={2015},
  note={arXiv:1512.01274}
}

• arXiv: 1512.01274