Compressive Transformer Eval

compressive-transformer-eval

Compressive Transformers for Long-Range Sequence Modelling — Jack W. Rae et al. (2019) (arXiv:1911.05507, 2019)

What this evaluates

Evaluates the ability of transformer-based architectures to model long-range dependencies efficiently by compressing past hidden states into a fixed-size memory. It probes sequence modeling capabilities across text, audio, and visual domains, measuring how well compressed representations preserve salient information for next-token prediction and task completion.

Datasets

• Enwiki8 — total 100000000; splits: train (90000000), val (5000000), test (5000000)
• WikiText-103 — total ?; splits: test (-1)
• PG-19 — total ?; splits: test (-1)
• DMLab-30 (rooms_select_nonmatching_object) — total ?; splits: test (-1)

Metrics

• perplexity (primary) — range: other
  • exp(mean(negative_log_likelihood(gold_tokens, predictions))). Measures the effective branching factor of the model's probability distribution over the next token.
• bits-per-character — range: other
  • mean(negative_log_likelihood(gold_chars, predictions)) / log(2). Standard character-level language modeling metric where lower is better.

Input / output format

Input: Sequences of characters, subword tokens, audio waveforms, or visual observation frames depending on the domain.

Output: Probability distribution over the next token/character, or sampled token.

Scoring recipe

def compute_perplexity(log_probs):
    avg_nll = np.mean(log_probs)
    return np.exp(avg_nll)

def compute_bpc(log_probs):
    avg_nll = np.mean(log_probs)
    return avg_nll / np.log(2)

Common pitfalls

• Confusing character-level perplexity (Enwiki8) with word-level perplexity (WikiText-103/PG-19), which are not directly comparable due to different tokenization vocabularies.
• Including dynamic evaluation (test-time training) in main comparisons, which the authors explicitly exclude to ensure fair model-to-model baselines.
• Assuming higher compression rates always improve performance; the paper notes performance degrades if compression is too aggressive (e.g., rate 1 fails on RL tasks).

Evidence (verbatim from paper)

The proposed model achieves the new state-of-the-art on this dataset with 0.97 bits-per-character.

Citation

@misc{rae2019compressive,
  title={Compressive Transformers for Long-Range Sequence Modelling},
  author={Jack W. Rae et al. (2019)},
  year={2019},
  note={arXiv:1911.05507}
}

• arXiv: 1911.05507