Neural Gpu Algorithmic Eval

neural-gpu-algorithmic-eval

Neural GPUs Learn Algorithms — Kaiser et al. (2015) (arXiv:1511.08228, 2015)

What this evaluates

Evaluates a model's ability to learn algorithmic rules (arithmetic, sequence transformation) from short training sequences and generalize them to arbitrarily long inputs without explicit programming or architectural changes.

Datasets

• Neural GPU Algorithmic Tasks — total ?; splits: test (-1); repo https://github.com/tensorflow/models/tree/master/neural_gpu

Metrics

• fully_correct_output_rate (primary) — range: [0, 1]
  • Fraction of test cases where every single bit of the model's output sequence exactly matches the ground truth sequence. No partial credit is awarded.

Input / output format

Input: A sequence of discrete symbols: binary digits (0, 1), an operator (+ or ·), and a padding symbol (PAD). For non-arithmetic tasks, a sequence of bits possibly with padding.

Output: A sequence of binary digits (0s and 1s) representing the computed result or transformed sequence.

Scoring recipe

correct_count = 0
total_count = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct_count += 1
    total_count += 1
return correct_count / total_count

Common pitfalls

• Generalization is evaluated on lengths far exceeding training length (e.g., train on 20-bit, test on 2000-bit).
• The metric is strict exact-match; partial correctness on any bit yields a score of 0 for that instance.
• Tasks are procedurally generated on-the-fly rather than drawn from a static corpus.

Evidence (verbatim from paper)

We measure also the rate of fully correct output sequences and report the results in Table 1. The table shows the fraction of test cases for which every single bit of the model’s output is correct.

Citation

@misc{kaiser2015neural,
  title={Neural GPUs Learn Algorithms},
  author={Kaiser et al. (2015)},
  year={2015},
  note={arXiv:1511.08228}
}

• arXiv: 1511.08228