Data Product Discovery Eval

data-product-discovery-eval

From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text — Zhang et al. (2025) (arXiv:2510.21737, 2025)

What this evaluates

Evaluates a model's ability to retrieve relevant tables and text passages from a hybrid corpus to satisfy complex, multi-part analytical user requests (Data Product Requests). It probes multi-modal data integration and semantic clustering capabilities by requiring complete alignment between a request and its underlying data assets.

Datasets

• HybridQA — total ?; splits: train (4889), dev (2008), test (1986)
• TAT-QA — total ?; splits: train (1157), dev (211), test (245)
• ConvFinQA — total ?; splits: train (2498), dev (426), test (875)

Metrics

• Recall@20 — range: [0, 1]
  • Fraction of ground-truth tables or passages retrieved within the top 20 results. Considers partial matches for individual tables or passages.
• Recall@100 — range: [0, 1]
  • Fraction of ground-truth tables or passages retrieved within the top 100 results. Considers partial matches for individual tables or passages.
• Full Recall@100 (primary) — range: [0, 1]
  • Binary metric indicating whether all ground-truth tables and passages for a data product are retrieved within the top 100 results. Considers only complete matches.

Input / output format

Input: A natural-language Data Product Request (DPR) describing a complex analytical need requiring multiple tables and passages.

Output: A ranked list of retrieved tables and text passages from the corpus.

Scoring recipe

def compute_recall(retrieved_top_k, ground_truth, k=100):
    retrieved_set = set(retrieved_top_k[:k])
    gt_set = set(ground_truth)
    if not gt_set: return 0.0
    return len(retrieved_set & gt_set) / len(gt_set)

def compute_full_recall(retrieved_top_100, ground_truth):
    return 1.0 if set(ground_truth).issubset(set(retrieved_top_100)) else 0.0

Common pitfalls

• Partial matches are acceptable for table/text Recall@K, but Full Recall@100 requires retrieving every single ground-truth table and passage for a data product.
• The benchmark relies on LLM-generated DPRs and LLM-as-a-judge validation, so evaluation results may reflect generation/judge biases rather than pure retrieval capability.
• Ground-truth alignments are pruned if judges flag tables as irrelevant, which can shrink the effective test set size per instance.

Evidence (verbatim from paper)

Table and text retreival alone consider partial matches, while under data product, Full Recall $@$ 100 considers only complete matches.

Citation

@misc{zhang2025dataproductbenchmark,
  title={From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.21737}
}

• arXiv: 2510.21737