Mudabench Eval

mudabench-eval

Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA — Li et al. (2026) (arXiv:2604.22239, 2026)

What this evaluates

MuDABench probes multi-document analytical QA capabilities, requiring models to synthesize quantitative insights and perform inter-document reasoning across large collections of heterogeneous financial documents. It specifically tests cross-document filtering, conditional information extraction, and multi-step numerical computation beyond standard single-document retrieval.

Datasets

• MuDABench — total ?; splits: test (-1); repo https://github.com/Zhanli-Li/MuDABench

Metrics

• final-answer accuracy (primary) — range: [0, 1]
  • Accuracy_final = (1/|Q|) * sum(T_i), where T_i is 1 if the model's predicted answer is semantically equivalent to the gold answer (judged by an LLM), else 0.
• process accuracy — range: [0, 1]
  • Accuracy_process = (1/|Q|) * sum(P_i). For standard RAG, P_i uses conservative coverage min(C_i, 1-E_i) based on LLM-judged semantic matches to gold facts. For document-grounded workflows, P_i is cell-wise accuracy |C_hat_i|/|C_i| on aligned rows.
• full accuracy — range: [0, 1]
  • Accuracy_full = (1/|Q|) * sum(m_i * T_i), where m_i is 1 if process accuracy P_i equals 1, and T_i is 1 if the final answer is correct.

Input / output format

Input: A natural language question $Q_j$ and a collection of $k$ financial documents (annual reports, announcements, ESG reports) with associated metadata (ticker, fiscal year, document type).

Output: A final answer $A_i$ (numerical or textual), and optionally intermediate extracted facts or table cells.

Scoring recipe

def score_final(gold_ans, pred_ans):
    return 1 if llm_judge_semantically_equivalent(pred_ans, gold_ans) else 0

def score_process_rag(gold_facts, pred_facts):
    C = len(intersect_semantically(pred_facts, gold_facts)) / len(gold_facts)
    E = count_missing_or_wrong(pred_facts, gold_facts) / len(gold_facts)
    return min(C, 1 - E)

def score_process_cell(gold_cells, pred_cells):
    return len(intersect_exact(pred_cells, gold_cells)) / len(gold_cells)

# Aggregate
final_acc = mean(score_final(g, p) for g, p in dataset)
process_acc = mean(score_process_rag(g, p) for g, p in dataset)  # or cell variant
full_acc = mean((process_score == 1.0) * score_final(g, p) for g, p in dataset)

Common pitfalls

• Process coverage can be less reliable when equivalent evidence can be expressed in multiple non-atomic fact forms, leading to potential underestimation of intermediate extraction quality.
• The evaluation relies heavily on an LLM-as-a-judge for semantic equivalence, which may overestimate coverage or introduce inconsistency compared to exact string matching.
• Standard RAG and document-grounded workflows require different scoring formulas (coverage vs. cell-wise accuracy); applying the wrong recipe yields invalid process accuracy scores.

Evidence (verbatim from paper)

We evaluate each system with three metrics: process accuracy, final-answer accuracy, and full accuracy. Among them, final-answer accuracy is our primary end-task metric, while process accuracy is mainly used as a diagnostic signal for intermediate extraction quality. We note that process coverage can be less reliable when equivalent evidence can be expressed in multiple non-atomic fact forms. Final-answer accuracy. For each question $Q_{i}\in\mathcal{Q}$, let $A_{i}$ be the gold final answer and $\hat{A}{i}$ be the model prediction. Let $T{i}\in{0,1}$ denote whether $\hat{A}{i}$ is semantically equivalent to $A{i}$ (judged by an LLM): Accuracy_final = (1/|Q|) \sum_{i}T_{i.

Citation

@misc{li2026mudabench,
  title={Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.22239}
}

• arXiv: 2604.22239