Patenteb Eval

patenteb-eval

PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding — Ayaou et al. (2025) (arXiv:2510.22264, 2025)

What this evaluates

Evaluates patent text embedding models across 15 diverse tasks including symmetric/asymmetric retrieval, classification, paraphrase detection, and clustering. It specifically probes domain-specific challenges like cross-domain retrieval, fragment-to-document matching, and temporal citation dynamics.

Datasets

• PatenTEB — total 2057286; splits: train (1556751), val (181215), test (319320); repo https://github.com/iliass-y/patenteb

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Measures ranking quality by summing relevance scores discounted by position, normalized by the ideal DCG.
• Macro-F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score. Computes F1 for each class independently and takes the unweighted mean, treating all classes equally regardless of support.
• Pearson r (primary) — range: [-1, 1]
  • Pearson correlation coefficient. Measures the linear correlation between predicted similarity scores and ground-truth labels.
• V-measure (primary) — range: [0, 1]
  • V-measure clustering metric. Harmonic mean of homogeneity and completeness, evaluating how well clusters match ground-truth groups.

Input / output format

Input: Varies by task: (1) Retrieval: query text and candidate patent documents/fragments; (2) Classification: patent text or citation pairs; (3) Paraphrase: pairs of patent segments; (4) Clustering: sets of patent families.

Output: Varies: (1) Retrieval: ranked list of candidates; (2) Classification: predicted class label; (3) Paraphrase: similarity score or binary label; (4) Clustering: cluster assignments.

Scoring recipe

def evaluate(predictions, gold, task_type):
    if task_type == 'retrieval':
        return ndcg_at_k(predictions, gold, k=10)
    elif task_type == 'classification':
        return macro_f1(predictions, gold)
    elif task_type == 'paraphrase':
        return pearson_correlation(predictions, gold)
    elif task_type == 'clustering':
        return v_measure_score(predictions, gold)

Common pitfalls

• Hard negative mining strictly excludes citation-connected candidates and enforces domain-specific selection to prevent models from relying on superficial domain cues.
• Asymmetric retrieval tasks apply deterministic fragment removal to targets to prevent trivial lexical matching between query and document.
• Clustering tasks are evaluation-only with no training data, requiring unsupervised embedding quality.
• Splits are stratified by IPC3 code to ensure domain balance and prevent patent family leakage across train/val/test.

Evidence (verbatim from paper)

Table 2. Summary of dataset statistics by task family. NDCG@10 is used for retrieval tasks, Pearson correlation for paraphrase, Macro-F1 for classification (with 20% few-shot training), and V-measure for clustering.

Citation

@misc{ayaou2025patenteb,
  title={PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding},
  author={Ayaou et al. (2025)},
  year={2025},
  note={arXiv:2510.22264}
}

• arXiv: 2510.22264