Loong Eval

loong-eval

SPD-RAG: Sub-Agent Per Document Retrieval-Augmented Generation — Akay et al. (2026) (arXiv:2603.08329, 2026)

What this evaluates

Probes long-context multi-document question answering by requiring models to synthesize evidence from all provided documents (10K–250K+ tokens) across financial reports and academic papers. It tests information extraction, comparison, clustering, and chain-of-reasoning capabilities in heterogeneous, document-level agentic retrieval settings.

Datasets

• Loong — total 102; splits: test (102)

Metrics

• Avg Score (primary) — range: [0, 100]
  • LLM-judged score (0–100) measuring the degree to which the predicted answer accurately covers the gold answer, following the Loong evaluation protocol. Evaluated using GPT-5 as the judge.
• Perfect Rate (PR%) — range: percent
  • Percentage of queries where the system receives a perfect score of 100.
• Avg Token Usage — range: other
  • Mean per-query total (input + output) token usage.
• Avg Cost (USD) — range: other
  • Mean per-query API cost, computed from token counts × per-token pricing.
• Avg Latency (s) — range: other
  • Mean per-query wall-clock time.

Input / output format

Input: A set of documents (average 11, ranging 10K to >250K tokens) in English or Chinese, along with a question requiring synthesis of evidence from all documents. Task types: Spotlight Locating, Comparison, Clustering, or Chain of Reasoning.

Output: A synthesized answer to the question.

Scoring recipe

def compute_avg_score(predictions, gold_answers, questions):
    scores = []
    for pred, gold, q in zip(predictions, gold_answers, questions):
        # GPT-5 judge evaluates coverage of gold answer on 0-100 scale
        score = llm_judge_gpt5(
            prompt=f"Question: {q}\nGold Answer: {gold}\nPredicted Answer: {pred}",
            system="Score 0-100 based on how accurately the prediction covers the gold answer."
        )
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• The evaluation uses GPT-5 as the judge instead of the original GPT-4, so raw scores are not perfectly 1-to-1 comparable with the originally published Loong leaderboard.
• The benchmark only evaluates on the English and Set 4 (200k-250k tokens) subset, not the full Loong benchmark.
• All documents are guaranteed relevant ("Leave No Document Behind"), so retrieval must cover all documents rather than filtering irrelevant ones.

Evidence (verbatim from paper)

We report the following metrics, following the Loong evaluation protocol: Avg Score: An LLM-judged score (0–100) measuring the degree to which the predicted answer accurately covers the gold answer, following the evaluation protocol of Loong (Wang et al., 2024a). Because the original legacy GPT-4 model API is unavailable for new scaled evaluations, we adopt GPT-5 as the evaluator.

Citation

@misc{akay2026spd_rag,
  title={SPD-RAG: Sub-Agent Per Document Retrieval-Augmented Generation},
  author={Akay et al. (2026)},
  year={2026},
  note={arXiv:2603.08329}
}

• arXiv: 2603.08329