Sparta Eval

sparta-eval

SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables — Park et al. (2026) (arXiv:2602.23286, 2026)

What this evaluates

Evaluates the ability of LLMs and hybrid QA systems to perform tree-structured, multi-hop reasoning over combined text and table data, including complex SQL operations like aggregation, grouping, ordering, and cross-modal retrieval.

Datasets

• SPARTA — total ?; splits: test (-1); repo https://github.com/pshlego/SPARTA

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of precision and recall over normalized answer matches. Precision is the fraction of predicted answers that match gold, recall is the fraction of gold answers matched by predictions.

Input / output format

Input: A natural language question, optionally accompanied by a set of relevant tables and text passages (Oracle setting) or provided with no evidence (Retrieval setting, requiring the model to first retrieve relevant tables/text).

Output: A natural language answer or executable query/response. The system evaluates the final answer string against gold answers using exact match and F1 scoring.

Scoring recipe

def compute_f1(preds, golds):
    em = sum(1 for p, g in zip(preds, golds) if normalize(p) == normalize(g)) / len(golds)
    p = sum(1 for p, g in zip(preds, golds) if normalize(p) in normalize(g)) / len(golds)
    r = sum(1 for p, g in zip(preds, golds) if normalize(g) in normalize(p)) / len(golds)
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return em, f1, p, r

Common pitfalls

• Models are evaluated under two distinct settings: Oracle (given gold tables/text) vs. Retrieval (must find evidence first). Performance drops significantly in the Retrieval setting.
• Standard single-hop QA benchmarks do not capture the difficulty of tree-structured multi-hop queries with aggregations/grouping, leading to misleadingly high baseline scores on older datasets.
• Table size matters: domains with thousands of rows (e.g., NBA, Movie) cause sharper performance degradation than smaller tables.

Evidence (verbatim from paper)

On SPARTA (Oracle), ODYSSEY with GPT-5 achieves an average F1 score of 35.6% across all domains, representing a sharp 33.9-point drop compared to its performance on HybridQA (69.5%). We evaluate SPARTA under two configurations: (1) SPARTA (Oracle), where models are given ground-truth tables and linked passages; and (2) SPARTA (Retrieval), where models must retrieve relevant content from the entire corpus.

Citation

@misc{park2026sparta,
  title={SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables},
  author={Park et al. (2026)},
  year={2026},
  note={arXiv:2602.23286}
}

• arXiv: 2602.23286