Dapfam Eval

dapfam-eval

DAPFAM: A Domain-Aware Family-level Dataset to benchmark cross domain patent retrieval — Ayaou et al. (2025) (arXiv:2506.22141, 2025)

What this evaluates

Evaluates cross-domain patent retrieval systems by measuring how well they rank relevant patent documents or passages when queries and targets share or lack overlapping IPC3 classifications.

Datasets

• DAPFAM — total ?; splits: ALL (-1), IN (-1), OUT (-1)

Metrics

• NDCG@100 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 100, averaged across queries within evaluation subsets.
• Recall@100 — range: [0, 1]
  • Recall at rank 100, measuring the proportion of relevant documents found in the top 100 results, averaged across queries.

Input / output format

Input: Patent query (Title, Abstract, Claims) and a corpus of patent documents or passages indexed for retrieval.

Output: Ranked list of up to 100 patent documents or passages per query.

Scoring recipe

def compute_ndcg_at_k(relevance_scores, k=100):
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(relevance_scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)[:k]
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

def compute_recall_at_k(relevance_scores, k=100):
    relevant_count = sum(1 for r in relevance_scores if r > 0)
    return sum(1 for r in relevance_scores[:k] if r > 0) / max(relevant_count, 1)

Common pitfalls

• Performance drops drastically on OUT-domain queries due to vocabulary divergence, so reporting only ALL/IN masks cross-domain failure.
• Passage length and aggregation strategy (avg_top3 vs maxP) must be tuned per backend and domain; fixed settings hurt performance.
• Execution time excludes preprocessing/indexing, so efficiency claims only reflect query-time scoring.

Evidence (verbatim from paper)

All results report NDCG@100 and Recall@100 averaged across queries within evaluation subsets.

Citation

@misc{ayaou2025dapfam,
  title={DAPFAM: A Domain-Aware Family-level Dataset to benchmark cross domain patent retrieval},
  author={Ayaou et al. (2025)},
  year={2025},
  note={arXiv:2506.22141}
}

• arXiv: 2506.22141