Loombench Eval

loombench-eval

LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework — Tang et al. (2025) (arXiv:2507.04723, 2025)

What this evaluates

Evaluates long-context language models across 22 benchmarks and 140 tasks, probing capabilities like long-form generation, information retrieval, and reasoning over extended contexts. It also assesses the efficiency of inference acceleration and RAG augmentation methods.

Datasets

• LOOMBench — total ?; splits: test (-1); repo https://github.com/LCM-Lab/LOOM-Scope

Metrics

• timing_results — range: seconds
  • Wall-clock inference time measured in seconds for processing 128K-length contexts or full task sets under different acceleration methods.
• task_accuracy (primary) — range: [0, 1]
  • Task-specific accuracy or performance score computed per subtask, as detailed in Appendix B.2 of the paper.

Input / output format

Input: Long-context prompts (up to 128K tokens) with task instructions, optionally augmented with RAG retrieval chunks (16K) or processed via acceleration method configurations.

Output: Model-generated text responses or predictions for each task, plus recorded inference latency for acceleration benchmarks.

Scoring recipe

for task in LOOMBench.tasks:
    samples = sample(task.instances, k=15)
    predictions = []
    for sample in samples:
        pred = model.generate(sample.context + sample.prompt)
        predictions.append(pred)
    task_score = compute_task_metric(predictions, sample.gold)
    results[task] = task_score
# For acceleration methods:
latency = measure_inference_time(model, context_length=128000, batch_size=8)

Common pitfalls

• Inconsistent prompt templates and inference settings across benchmarks lead to non-comparable results, which LOOM-Scope standardizes.
• Rule-based RAG (BM25) may underperform direct long-context models if retrieval chunking and routing are not optimized.
• Evaluation time and GPU memory requirements scale drastically with context length without acceleration methods like KV-cache pruning.

Evidence (verbatim from paper)

We evaluate on LOOMBench with three settings: (1) naive LCLM with HF_Models server, (2) RAG, and (3) inference acceleration methods. ... sampling 15 data instances per subtask under each method’s official configuration: the Native Transformer (FlashAttention implementation) used batch size 1, while all acceleration methods used batch size 8. The timing results for the methods tested on 40GB A100 and H20 GPUs are shown in Figure[7].

Citation

@misc{tang2025loomscope,
  title={LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2507.04723}
}

• arXiv: 2507.04723