Framework Latency Eval

framework-latency-eval

Comparative Study of Deep Learning Software Frameworks — Bahrampour et al. (2015) (arXiv:1511.06435, 2015)

What this evaluates

Evaluates the computational efficiency and hardware utilization of five deep learning frameworks (Caffe, Neon, TensorFlow, Theano, Torch) across standard neural network architectures on CPU and GPU hardware.

Datasets

• MNIST — total ?; splits: train (-1), test (-1)
• ImageNet — total ?; splits: train (-1), test (-1)
• IMDB — total ?; splits: train (-1), test (-1)

Metrics

• forward pass time (ms) (primary) — range: other
  • Average processing time in milliseconds over 20–1000 iterations after warm-up runs. Data loading and preprocessing times are explicitly excluded.
• gradient computation time (ms) — range: other
  • Average processing time in milliseconds for the backward pass over 20–1000 iterations after warm-up runs. Data loading and preprocessing times are explicitly excluded.
• peak GPU memory usage (GB) — range: other
  • Maximum GPU RAM consumed during the forward and backward pass execution for a given batch size.

Input / output format

Input: Neural network architecture definitions and dataset batches (images or text reviews) configured with fixed batch sizes (64 or 256).

Output: Timings in milliseconds for forward pass and gradient computation, and peak GPU memory consumption in gigabytes.

Scoring recipe

def compute_latency(model, batch, num_iters=100):
    for _ in range(20): model.forward(batch); model.zero_grad()
    times = []
    for _ in range(num_iters):
        t0 = time.time()
        out = model.forward(batch)
        loss = compute_loss(out, batch.labels)
        loss.backward()
        times.append(time.time() - t0)
    return sum(times) / len(times)

Common pitfalls

• Data loading and preprocessing times are explicitly excluded from the reported forward and backward pass times.
• Performance is highly dependent on the underlying convolution library (e.g., cuDNN v2 vs v3, conv-fft, fbcunn), making results library-specific rather than framework-agnostic.
• Batch size and kernel size constraints vary by framework (e.g., Neon requires multiples of 4 and 32 on GPU), which can alter architecture comparability.

Evidence (verbatim from paper)

Table 3 shows the averaged processing time for gradient computation as well as the time for a forward step obtained by the five frameworks on both CPU and GPU using batch size of 64. The timings reported here are average of 20-1000 iterations and are controlled to have small standard deviations. ... the time required for data loading and processing (mean normalization) in each batch is excluded from time of forward and backward steps in all our experiments. We also report the peak GPU memory consumption to illustrate the efficacy of the frameworks in implementing deep networks.

Citation

@misc{bahrampour2015comparative,
  title={Comparative Study of Deep Learning Software Frameworks},
  author={Bahrampour et al. (2015)},
  year={2015},
  note={arXiv:1511.06435}
}

• arXiv: 1511.06435