# Loombench Eval

> Evaluates long-context language models across 22 benchmarks and 140 tasks, probing capabilities like long-form generation, information retrieval, and reasoning over extended contexts. It also assesses the efficiency of inference acceleration and RAG augmentation methods. Use when the user wants to benchmark on LOOMBench, or asks about evaluating this task. Reports task_accuracy.

- Skill: `qhjqhj00/loombench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/loombench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/loombench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/loombench-eval

---


# loombench-eval

> LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework — Tang et al. (2025) (arXiv:2507.04723, 2025)

## What this evaluates

Evaluates long-context language models across 22 benchmarks and 140 tasks, probing capabilities like long-form generation, information retrieval, and reasoning over extended contexts. It also assesses the efficiency of inference acceleration and RAG augmentation methods.

## Datasets

- **LOOMBench** — total ?; splits: test (-1); repo https://github.com/LCM-Lab/LOOM-Scope

## Metrics

- `timing_results` — range: seconds
  - Wall-clock inference time measured in seconds for processing 128K-length contexts or full task sets under different acceleration methods.
- `task_accuracy` **(primary)** — range: [0, 1]
  - Task-specific accuracy or performance score computed per subtask, as detailed in Appendix B.2 of the paper.

## Input / output format

**Input**: Long-context prompts (up to 128K tokens) with task instructions, optionally augmented with RAG retrieval chunks (16K) or processed via acceleration method configurations.

**Output**: Model-generated text responses or predictions for each task, plus recorded inference latency for acceleration benchmarks.

## Scoring recipe

```python
for task in LOOMBench.tasks:
    samples = sample(task.instances, k=15)
    predictions = []
    for sample in samples:
        pred = model.generate(sample.context + sample.prompt)
        predictions.append(pred)
    task_score = compute_task_metric(predictions, sample.gold)
    results[task] = task_score
# For acceleration methods:
latency = measure_inference_time(model, context_length=128000, batch_size=8)
```

## Common pitfalls

- Inconsistent prompt templates and inference settings across benchmarks lead to non-comparable results, which LOOM-Scope standardizes.
- Rule-based RAG (BM25) may underperform direct long-context models if retrieval chunking and routing are not optimized.
- Evaluation time and GPU memory requirements scale drastically with context length without acceleration methods like KV-cache pruning.

## Evidence (verbatim from paper)

> We evaluate on LOOMBench with three settings: (1) naive LCLM with HF_Models server, (2) RAG, and (3) inference acceleration methods. ... sampling 15 data instances per subtask under each method’s official configuration: the Native Transformer (FlashAttention implementation) used batch size 1, while all acceleration methods used batch size 8. The timing results for the methods tested on 40GB A100 and H20 GPUs are shown in Figure[7].

## Citation

```bibtex
@misc{tang2025loomscope,
  title={LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2507.04723}
}
```

- arXiv: 2507.04723