Mlperf Input Pipeline Eval

mlperf-input-pipeline-eval

tf.data: A Machine Learning Data Processing Framework — Murray et al. (2021) (arXiv:2101.12127, 2021)

What this evaluates

This evaluation protocol measures the throughput and latency of machine learning input data pipelines across standard computer vision and NLP benchmarks. It probes how efficiently a data processing framework can ingest, transform, and feed batches to a training loop compared to sequential baselines and competing systems.

Datasets

• ImageNet — total ?; splits: train (-1), val (-1), test (-1)
• COCO — total ?; splits: train (-1), val (-1), test (-1)
• WMT16 — total ?; splits: train (-1), val (-1), test (-1)
• WMT17 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• epoch duration (primary) — range: seconds
  • Wall-clock time in seconds to process one full epoch of input data through the pipeline without model training computation.
• time to convergence — range: seconds
  • Wall-clock time in seconds to train the model until it reaches a predefined target accuracy metric.

Input / output format

Input: Raw dataset files (images, text sequences) passed through a declarative tf.data pipeline with transformations like map, interleave, and prefetch.

Output: Preprocessed batches of data fed sequentially to the model training loop.

Scoring recipe

def evaluate_pipeline(pipeline_config, baseline_config, target_accuracy):
    baseline_time = run_pipeline_in_loop(baseline_config, epochs=1)
    optimized_time = run_pipeline_in_loop(pipeline_config, epochs=1)
    speedup = baseline_time / optimized_time
    convergence_time = train_model_until_accuracy(pipeline_config, target=target_accuracy)
    return {
        'epoch_duration': optimized_time,
        'throughput_speedup': speedup,
        'time_to_convergence': convergence_time
    }

Common pitfalls

• Baseline pipelines may still utilize implicit TensorFlow parallelism for user-defined map functions, inflating the baseline performance and reducing measured speedup.
• Cross-system comparisons often conflate input pipeline speed with full training stack performance due to differing hardware, frameworks, and expert tuning levels.
• Auto-tuned configurations match expert-tuned performance but may require longer warm-up periods to converge on optimal buffer sizes and parallelism degrees.

Evidence (verbatim from paper)

Figure 7 shows the mean duration of a single epoch, normalized to the epoch duration of the baseline, which does not use any tf.data performance-related features.

Citation

@misc{murray2021tfdata,
  title={tf.data: A Machine Learning Data Processing Framework},
  author={Murray et al. (2021)},
  year={2021},
  note={arXiv:2101.12127}
}

• arXiv: 2101.12127