Test Accuracy

test-accuracy

Revisiting Distributed Synchronous SGD — Chen et al. (2016) (arXiv:1604.00981, 2016)

What this evaluates

Evaluates the convergence speed and final test performance of distributed synchronous versus asynchronous stochastic gradient descent algorithms. It probes whether backup workers in synchronous training can mitigate stragglers without degrading accuracy due to gradient staleness.

Datasets

• ImageNet — total ?; splits: train (-1), test (-1)
• CIFAR-10 — total ?; splits: train (-1), test (-1)

Metrics

• test accuracy (primary) — range: percent
  • Percentage of correctly classified images on the validation/test set. Computed as (correct predictions / total predictions) * 100.
• epochs to converge — range: int
  • Number of training epochs required to reach a predefined target test precision threshold.
• time to converge — range: hours
  • Wall-clock time in hours required to reach the target test precision or NLL threshold.
• negative log likelihood (NLL) — range: other
  • Test negative log likelihood for the PixelCNN generative model, where lower values indicate better density estimation.

Input / output format

Input: Distributed training of Inception on ImageNet and PixelCNN on CIFAR-10 using TensorFlow, with varying total workers (N) and backup workers (b) from 1 to 212.

Output: Convergence curves and final reported values for test precision/accuracy, epochs to reach target precision, wall-clock time to converge, and test NLL.

Scoring recipe

def compute_metrics(predictions, gold, target_precision=0.77):
    accuracy = (predictions == gold).mean() * 100
    epochs = count_epochs_until(accuracy >= target_precision)
    time = wall_clock_time_until(accuracy >= target_precision)
    return accuracy, epochs, time

Common pitfalls

• Focusing only on early-phase convergence speed can be misleading, as faster convergence may lead to poorer local optima with lower final accuracy.
• Comparing epoch counts across synchronous and asynchronous methods without accounting for the fact that an epoch represents different amounts of computation in each setting.

Evidence (verbatim from paper)

We are interested in two metrics of comparison for our empirical validation: (1) test error or accuracy, and (2) speed of convergence.

Citation

@misc{chen2016revisiting,
  title={Revisiting Distributed Synchronous SGD},
  author={Chen et al. (2016)},
  year={2016},
  note={arXiv:1604.00981}
}

• arXiv: 1604.00981