Long Range Arena Eval

long-range-arena-eval

Long Range Arena: A Benchmark for Efficient Transformers — Tay et al. (2020) (arXiv:2011.04006, 2020)

What this evaluates

Evaluates efficient Transformer architectures on long-context sequence modeling tasks spanning text, images, and structured data. Probes capabilities in hierarchical reasoning, spatial navigation, and retrieval over sequences up to 16K tokens.

Datasets

• ListOps — total ?; splits: train (-1), test (-1)
• Text Classification — total ?; splits: train (-1), test (-1)
• Retrieval — total ?; splits: train (-1), test (-1)
• Image Classification — total ?; splits: train (-1), test (-1)
• Pathfinder / Path-X — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of instances.
• steps per second — range: other
  • Throughput metric measuring the number of training steps completed per second on the specified hardware.
• peak memory usage — range: other
  • Maximum GPU/TPU memory consumed per device during evaluation, measured in gigabytes.

Input / output format

Input: Task-specific sequences: mathematical expression lists, byte-level character sequences, image patches, or path graphs. Sequence lengths typically range from 1K to 16K tokens.

Output: Discrete class labels corresponding to the task (e.g., 10-way classification for ListOps, binary/multi-class for others).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Results are based on fixed hyperparameters across all models to ensure fairness, so they do not represent optimally tuned performance for any single architecture.
• Efficiency metrics (speed/memory) are highly dependent on the specific hardware (4x4 TPU V3) and implementation details used in the benchmark.
• Models frequently overfit training data but show a large generalization gap on test sets, particularly for image classification tasks.

Evidence (verbatim from paper)

Byte-level classification is shown to be difficult and challenging especially when no pretraining or contextual embeddings are used. The best model only obtains 65.90 accuracy.

Citation

@misc{tay2020longrangearena,
  title={Long Range Arena: A Benchmark for Efficient Transformers},
  author={Tay et al. (2020)},
  year={2020},
  note={arXiv:2011.04006}
}

• arXiv: 2011.04006