Training Speed Eval

training-speed-eval

ScaleFold: Reducing AlphaFold Initial Training Time to 10 Hours — Zhu et al. (2024) (arXiv:2404.11068, 2024)

What this evaluates

Evaluates the training efficiency and scalability of AlphaFold-like models on GPU clusters. It measures per-step execution time, overall wall-clock training duration, and convergence speed across different hardware configurations and optimization techniques.

Datasets

• OpenFold dataset — total ?; splits: train (-1)

Metrics

• step time (primary) — range: seconds
  • Wall-clock time to complete one training iteration (forward pass, backward pass, and optimizer step).
• time to train — range: minutes
  • Total wall-clock time required to complete the full pretraining run or reach a target convergence threshold.
• avg_lddt_ca — range: [0, 1]
  • Average Local Distance Difference Test for C-alpha atoms, computed during training to monitor structural prediction quality.

Input / output format

Input: Mini-batches of protein sequence and structural data from the OpenFold dataset, processed through the AlphaFold2-like model architecture.

Output: Logged training step time (seconds), cumulative training time (minutes/hours), and intermediate avg_lddt_ca scores at specified training intervals.

Scoring recipe

def compute_metrics(logs, baseline_logs):
    step_time = logs['end_time'] - logs['start_time']
    speedup = baseline_logs['step_time'] / step_time
    time_to_train = logs['total_wall_clock'] / 60.0
    converged = logs['avg_lddt_ca'] >= 0.8
    return step_time, speedup, time_to_train, converged

Common pitfalls

• Confusing the DAP-1 (NoDAP) baseline with the reference model baseline; reported speedup ratios differ significantly depending on which baseline is used.
• Overlooking that avg_lddt_ca is a training convergence metric, not a final held-out test set evaluation; the paper does not report final test accuracy.
• Assuming a fixed global batch size across all experiments; the training protocol switches from batch size 128 to 256 after the first 5000 steps.

Evidence (verbatim from paper)

We compared the step time of our implementation to public OpenFold and FastFold. The results, as presented in Figure 7, demonstrating that the training performance of ScaleFold outperforms others. ... On H100, the step time of ScaleFold DAP-1 (NoDAP), DAP-2, DAP-4 and DAP-8 are 1.80s, 1.12s, 0.75s and 0.65s, respectively. ... Training metric avg_lddt_ca must exceed 0.8 before first 5000 training steps. ... The whole AlphaFold pretraining requires 50000~60000 steps to reach 0.9 avg_lddt_ca, which takes < 10 hours.

Citation

@misc{zhu2024scalefold,
  title={ScaleFold: Reducing AlphaFold Initial Training Time to 10 Hours},
  author={Zhu et al. (2024)},
  year={2024},
  note={arXiv:2404.11068}
}

• arXiv: 2404.11068