Pneuma Table Retrieval Eval

pneuma-table-retrieval-eval

Pneuma: Leveraging LLMs for Tabular Data Representation and Retrieval in an End-to-End System — Balaka et al. (2025) (arXiv:2504.09207, 2025)

What this evaluates

This evaluation probes a retrieval system's ability to identify relevant tabular datasets from a corpus given natural language questions. It measures retrieval accuracy via hit rate, alongside system efficiency metrics including query throughput, offline preparation time, and storage footprint across diverse real-world and benchmark datasets.

Datasets

• ChEMBL — total ?; splits: test (-1)
• Adventure Works — total ?; splits: test (-1)
• Public BI — total ?; splits: test (-1)
• Chicago Open Data — total ?; splits: test (-1)
• FeTaQA — total ?; splits: test (-1)
• BIRD — total ?; splits: test (-1)

Metrics

• hit rate@k (primary) — range: percent
  • Proportion of questions for which at least one relevant (gold) table appears in the top-k retrieved documents. Multiple gold tables per question are allowed; finding any one counts as a hit.
• query throughput — range: other
  • Number of queries processed per unit time, measured by recording total time to answer 100 benchmark questions and averaging over 10 runs.
• offline preparation time — range: other
  • Wall-clock time required to ingest data, generate table representations (e.g., schema narrations, embeddings), and build the search index.
• storage footprint — range: other
  • Disk space required to store the indexed representations, vectors, and auxiliary data structures.

Input / output format

Input: Natural language question.

Output: Ranked list of candidate tables retrieved by the system (top-k tables).

Scoring recipe

def compute_hit_rate(retrieved_tables, gold_tables, k):
    correct = 0
    for q in range(len(retrieved_tables)):
        top_k = retrieved_tables[q][:k]
        if any(table in top_k for table in gold_tables[q]):
            correct += 1
    return (correct / len(retrieved_tables)) * 100

Common pitfalls

• SOLO's hyperparameter k denotes the number of tables to return, making it incomparable to other baselines at k=5.
• BIRD benchmark requires distinguishing between 'original' (single gold table per question) and 'annotated' (multiple valid tables) splits, which significantly impacts hit rate calculations.
• LLAMAINDEX and SOLO face severe memory/storage constraints on large datasets (e.g., Chicago, BIRD), often requiring row limits or causing out-of-memory failures that must be accounted for when comparing efficiency.

Evidence (verbatim from paper)

We use hit rate to evaluate PNEUMA's performance in identifying tables relevant to the given questions. In our benchmarks, while a question may have multiple relevant tables, any one of these tables is sufficient to answer the question fully. Therefore, the hit rate is defined as the proportion of questions for which at least one relevant table appears in the top- $k$ retrieved documents.

Citation

@misc{balaka2025pneuma,
  title={Pneuma: Leveraging LLMs for Tabular Data Representation and Retrieval in an End-to-End System},
  author={Balaka et al. (2025)},
  year={2025},
  note={arXiv:2504.09207}
}

• arXiv: 2504.09207