Ruler Eval

ruler-eval

RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh et al. (2024) (arXiv:2404.06654, 2024)

What this evaluates

This benchmark evaluates long-context language models' ability to retrieve, trace, aggregate, and answer questions across varying context lengths and task complexities. It probes whether models genuinely attend to injected information or rely on parametric knowledge and context copying as sequence length increases.

Datasets

• RULER — total ?; splits: test (-1); repo https://github.com/hsiehjackson/RULER

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • Percentage of test instances where the model's generated output exactly matches the gold answer string. Evaluated across all task categories (NIAH variants, VT, CWE, FWE, QA).

Input / output format

Input: A long text sequence containing a 'haystack' of distractor text (e.g., essays, noise sentences, or random paragraphs), with injected key-value pairs, variable chains, or golden paragraphs. A query or question is appended at the end.

Output: The model must generate the retrieved values, tracked variable names, top-K frequent words, or the answer to the question, typically as a short string or list of tokens.

Scoring recipe

def compute_exact_match_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return correct / len(golds)

Common pitfalls

• Models may rely on parametric knowledge or memorization rather than attending to the long context, especially as length increases.
• Hard distractors in multi-key NIAH tasks can cause retrieval failures if the model confuses needles with noise.
• Performance degradation is often non-monotonic and highly sensitive to specific context lengths and task configurations, making single-point comparisons misleading.

Evidence (verbatim from paper)

Evaluation examples in Ruler are automatically generated based on input configurations (see Table 2) that define the length and complexity of each input. The objective is to return all variable names pointing to the same value V.

Citation

@misc{hsieh2024ruler,
  title={RULER: What's the Real Context Size of Your Long-Context Language Models?},
  author={Cheng-Ping Hsieh et al. (2024)},
  year={2024},
  note={arXiv:2404.06654}
}

• arXiv: 2404.06654