Landmark Attention Eval

landmark-attention-eval

Landmark Attention: Random-Access Infinite Context Length for Transformers — Mohtanami et al. (2023) (arXiv:2305.16300, 2023)

What this evaluates

Evaluates a transformer variant's ability to retrieve relevant past context blocks and maintain language modeling performance over extended sequence lengths. It probes long-range dependency retention and random-access memory retrieval capabilities compared to standard and recurrent transformer baselines.

Datasets

• PG-19 — total ?; splits: test (-1)
• arXiv math papers — total ?; splits: test (-1)
• RedPajama (subset) — total ?; splits: fine-tuning (-1); repo https://github.com/togethercomputer/RedPajama-Data

Metrics

• perplexity (primary) — range: other
  • Standard language modeling metric: exp(mean(-log_softmax(logits) * one_hot(targets))). Lower values indicate better performance.
• pass key accuracy — range: [0, 1]
  • Proportion of prompts where the model correctly outputs the hidden integer pass key within the first 100 generated tokens. Averaged over 50 random pass key generations and positions.

Input / output format

Input: Text segments divided into chunks of length l_local (e.g., 250 tokens) with landmark tokens inserted. For retrieval evaluation: prompts containing a prefix filler, a randomly generated pass key at a random position, and a suffix filler.

Output: Next-token probability distribution (for perplexity) or the generated integer pass key (for retrieval accuracy).

Scoring recipe

# Perplexity
perplexity = exp(mean(-log_softmax(logits) * one_hot(targets)))

# Pass key accuracy
correct = 0
for prompt in prompts:
    generated = model.generate(prompt, max_new_tokens=100)
    predicted_key = extract_first_integer(generated)
    if predicted_key == prompt.pass_key:
        correct += 1
accuracy = correct / len(prompts)

Common pitfalls

• Confusing local context length (l_local) with total effective context length, which includes retrieved blocks and landmark tokens.
• Assuming retrieval accuracy is computed over a fixed dataset split; it is actually averaged over 50 random generations of the pass key and its position.
• Overlooking that evaluating contexts >2048 tokens requires KV cache offloading to CPU, which affects memory but not the metric itself.

Evidence (verbatim from paper)

Table 1 presents the perplexity of the trained models under various inference settings. ... compute the accuracy of generating the correct pass key (as the first integer within the first 100 generated tokens).

Citation

@misc{mohtanami2023landmark,
  title={Landmark Attention: Random-Access Infinite Context Length for Transformers},
  author={Mohtanami et al. (2023)},
  year={2023},
  note={arXiv:2305.16300}
}

• arXiv: 2305.16300