Recall Throughput Eval

recall-throughput-eval

Simple linear attention language models balance the recall-throughput tradeoff — Simran Arora et al. (2024) (arXiv:2402.18668, 2024)

What this evaluates

Evaluates language models on associative recall, information extraction, and question answering from long contexts, while measuring generation throughput and language modeling perplexity. It probes the tradeoff between memory efficiency, recall accuracy, and inference speed across synthetic and real-world benchmarks.

Datasets

• Pile — total ?; splits: test (-1)
• SWDE — total ?; splits: test (-1)
• FDA — total ?; splits: test (-1)
• SQUAD — total ?; splits: test (-1)
• LM Eval Harness (SuperGLUE, ARC, PIQA, WinoGrande, HellaSwag, LAMBADA) — total ?; splits: test (-1)

Metrics

• perplexity (primary) — range: other
  • Exponential of the negative average log-likelihood of the next token given the context: exp(-1/N Σ log p(x_i)). Lower is better.
• accuracy — range: [0, 1]
  • Fraction of correctly predicted answers or extracted attributes out of the total number of instances.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall, computed at the token or exact-match level depending on the task.
• throughput — range: tokens/ms
  • Number of tokens generated or processed divided by the wall-clock time in milliseconds.

Input / output format

Input: Text sequences tokenized with GPT-2 BPE. For language modeling, autoregressive context windows. For IE/QA, prompts containing documents/passages followed by questions or extraction queries.

Output: Next-token probability distributions for perplexity. Extracted attribute values or direct answers for IE/QA tasks. Generated token sequences for throughput measurement.

Scoring recipe

def compute_metrics(predictions, gold, times):
    # Perplexity
    ppl = math.exp(-sum(torch.log_softmax(logits, dim=-1)[range(len(tokens))]) / len(tokens))
    # Accuracy & F1 for IE/QA
    preds = extract_answer_or_attribute(model_output)
    acc = (preds == gold).float().mean()
    f1 = compute_f1(preds, gold) # token-level or exact match
    # Throughput
    throughput = tokens_generated / (end_time - start_time) # tokens/ms
    return ppl, acc, f1, throughput

Common pitfalls

• Confusing prefill throughput (processing long context) with generation throughput (autoregressive decoding), which differ by orders of magnitude.
• Failing to use the exact GPT-2 BPE tokenizer for Pile evaluation, which changes tokenization and perplexity scores.
• Not separating the 'associative recall' (AR) slice from the 'other' slice when reporting Pile perplexity, as they measure distinct capabilities.

Evidence (verbatim from paper)

We report language model perplexity on the overall Pile test set as well as perplexity on two slices of the test set: associative recall tokens and other tokens... We report zero-shot performance on three recall-intensive tasks: information retrieval on SWDE and FDA as well as question answering on SQUAD. Finally, we report average performance on the set of LM Eval Harness common sense reasoning tasks used in gu2023mamba, details in [Appendix D].

Citation

@misc{arora2024simplelinearattention,
  title={Simple linear attention language models balance the recall-throughput tradeoff},
  author={Simran Arora et al. (2024)},
  year={2024},
  note={arXiv:2402.18668}
}

• arXiv: 2402.18668