Aibench Training Eval

aibench-training-eval

AIBench Training: Balanced Industry-Standard AI Training Benchmarking — Tang et al. (2020) (arXiv:2004.14690, 2020)

What this evaluates

Evaluates AI training workloads by measuring model complexity, computational cost, convergence rate, and micro-architectural behavior to assess benchmark diversity, repeatability, and cost against industry standards like MLPerf.

Datasets

• AIBench Training — total 19; splits: full (19)

Metrics

• convergent_rate (primary) — range: epochs
  • Number of training epochs required to reach a predefined target quality metric (e.g., accuracy, perplexity, WER, mAP) specific to each task.
• run_to_run_variation — range: percent
  • Coefficient of variation calculated as the standard deviation divided by the mean of the number of epochs (or iterations/time) across multiple independent runs under identical configurations.

Input / output format

Input: Model architecture, dataset, optimizer, loss function, batch size, and target quality threshold for a specific AI training task.

Output: Training logs containing epochs to convergence, total training time, per-epoch time, run-to-run variation coefficient, and micro-architectural utilization metrics.

Scoring recipe

def evaluate_benchmark(runs, target_quality):
    # Find epochs to reach target quality
    convergent_rate = min(r['epochs'] for r in runs if r['quality'] >= target_quality)
    # Calculate run-to-run variation (coefficient of variation)
    epochs = [r['epochs'] for r in runs]
    variation = (np.std(epochs) / np.mean(epochs)) * 100
    return convergent_rate, variation

Common pitfalls

• OpCounter tool underestimates FLOPs and learnable parameters for some operations, so reported model complexity may be lower than actual.
• Iteration-based models (e.g., BERT, DLRM) cannot use epoch-based convergence metrics; they require iteration or time-based tracking.
• Target quality thresholds are task-specific (accuracy, perplexity, WER, mAP, etc.), preventing direct cross-task numerical comparison without normalization.

Evidence (verbatim from paper)

We report the convergent rate—the cost of training a model. We use the total amount of learnable parameters, FLOPs of a single forward computation, and the number of epochs to achieve a target quality (e.g., accuracy) to characterize the above three characteristics, respectively. HPC AI500 uses the coefficient of variation(the standard deviation / mean) of the number of epochs to quantify the run-to-run variation

Citation

@misc{tang2020aibench,
  title={AIBench Training: Balanced Industry-Standard AI Training Benchmarking},
  author={Tang et al. (2020)},
  year={2020},
  note={arXiv:2004.14690}
}

• arXiv: 2004.14690