# Mlperf Input Pipeline Eval

> This evaluation protocol measures the throughput and latency of machine learning input data pipelines across standard computer vision and NLP benchmarks. It probes how efficiently a data processing framework can ingest, transform, and feed batches to a training loop compared to sequential baselines and competing systems. Use when the user wants to benchmark on ImageNet, COCO, WMT16, WMT17, or asks about evaluating this task. Reports epoch duration.

- Skill: `qhjqhj00/mlperf-input-pipeline-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mlperf-input-pipeline-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mlperf-input-pipeline-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mlperf-input-pipeline-eval

---


# mlperf-input-pipeline-eval

> tf.data: A Machine Learning Data Processing Framework — Murray et al. (2021) (arXiv:2101.12127, 2021)

## What this evaluates

This evaluation protocol measures the throughput and latency of machine learning input data pipelines across standard computer vision and NLP benchmarks. It probes how efficiently a data processing framework can ingest, transform, and feed batches to a training loop compared to sequential baselines and competing systems.

## Datasets

- **ImageNet** — total ?; splits: train (-1), val (-1), test (-1)
- **COCO** — total ?; splits: train (-1), val (-1), test (-1)
- **WMT16** — total ?; splits: train (-1), val (-1), test (-1)
- **WMT17** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `epoch duration` **(primary)** — range: seconds
  - Wall-clock time in seconds to process one full epoch of input data through the pipeline without model training computation.
- `time to convergence` — range: seconds
  - Wall-clock time in seconds to train the model until it reaches a predefined target accuracy metric.

## Input / output format

**Input**: Raw dataset files (images, text sequences) passed through a declarative tf.data pipeline with transformations like map, interleave, and prefetch.

**Output**: Preprocessed batches of data fed sequentially to the model training loop.

## Scoring recipe

```python
def evaluate_pipeline(pipeline_config, baseline_config, target_accuracy):
    baseline_time = run_pipeline_in_loop(baseline_config, epochs=1)
    optimized_time = run_pipeline_in_loop(pipeline_config, epochs=1)
    speedup = baseline_time / optimized_time
    convergence_time = train_model_until_accuracy(pipeline_config, target=target_accuracy)
    return {
        'epoch_duration': optimized_time,
        'throughput_speedup': speedup,
        'time_to_convergence': convergence_time
    }
```

## Common pitfalls

- Baseline pipelines may still utilize implicit TensorFlow parallelism for user-defined map functions, inflating the baseline performance and reducing measured speedup.
- Cross-system comparisons often conflate input pipeline speed with full training stack performance due to differing hardware, frameworks, and expert tuning levels.
- Auto-tuned configurations match expert-tuned performance but may require longer warm-up periods to converge on optimal buffer sizes and parallelism degrees.

## Evidence (verbatim from paper)

> Figure 7 shows the mean duration of a single epoch, normalized to the epoch duration of the baseline, which does not use any tf.data performance-related features.

## Citation

```bibtex
@misc{murray2021tfdata,
  title={tf.data: A Machine Learning Data Processing Framework},
  author={Murray et al. (2021)},
  year={2021},
  note={arXiv:2101.12127}
}
```

- arXiv: 2101.12127