qhjqhj00 modeltables-eval
ModelTables: A Corpus of Tables about Models — Dong et al. (2025) (arXiv:2512.16106, 2025)
What this evaluates
Evaluates the ability of retrieval systems to find relevant AI model tables from a heterogeneous corpus. It probes semantic understanding of structured data and cross-source table matching.
Datasets
- ModelTables — total ?; splits: test (-1); repo https://github.com/RJMillerLab/ModelTables
Metrics
P@1(primary) — range: percent- Precision at 1: returns 1 if the top-ranked retrieved table is relevant to the query, and 0 otherwise. Averaged across all queries.
Input / output format
Input: A query table (or model identifier) and a corpus of tables extracted from Hugging Face model cards, GitHub READMEs, arXiv papers, and Semantic Scholar.
Output: A ranked list of retrieved tables.
Scoring recipe
def compute_p_at_1(retrieved_tables, relevant_tables):
top_1 = retrieved_tables[0]
return 1.0 if top_1 in relevant_tables else 0.0
# Average over all queries
scores = [compute_p_at_1(q_retrieved, q_relevant) for q_retrieved, q_relevant in queries]
metric_value = sum(scores) / len(scores) * 100
Common pitfalls
- Heterogeneous table formats across sources (e.g., transposed layouts, merged headers) can break naive retrieval without augmentation.
- S2ORC tables are raw unstructured text requiring LLM reconstruction, introducing potential parsing noise.
- Placeholder arXiv links in Hugging Face templates must be explicitly filtered to avoid irrelevant tables.
Evidence (verbatim from paper)
It establishes multi-source ground truth for table relatedness via model card links, paper citations, and shared datasets, enabling evaluation of table search methods. Empirical results show table-based dense retrieval (66.5% P@1) outperforms union-based semantic retrieval (54.8% P@1) and metadata-hybrid retrieval (54.1%), highlighting the need for better semantic understanding of structured model knowledge.
Citation
@misc{dong2025modeltables,
title={ModelTables: A Corpus of Tables about Models},
author={Dong et al. (2025)},
year={2025},
note={arXiv:2512.16106}
}
- arXiv: 2512.16106