Hupd Eval

hupd-eval

The Harvard USPTO Patent Dataset: A Large-Scale, Well-Structured, and Multi-Purpose Corpus of Patent Applications — Suzgun et al. (2022) (arXiv:2207.04043, 2022)

What this evaluates

Evaluates NLP models on patent-related tasks including binary classification of patent acceptance, multi-class subject area classification using IPC codes, and abstractive summarization of patent claims or descriptions into abstracts.

Datasets

• Harvard USPTO Patent Dataset (HUPD) — total 4500000; splits: train (-1), test (-1); repo https://github.com/suzgunmirac/hupd

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted labels out of total instances. For the acceptance task, test sets are strictly balanced (50% accepted, 50% rejected), making 50% the baseline.
• TOP1 — range: [0, 1]
  • Top-1 accuracy for multi-class IPC classification; checks if the single highest probability prediction matches the actual label.
• TOP5 — range: [0, 1]
  • Top-5 accuracy; checks if the actual label is among the five classes with the highest predicted probabilities.
• ROUGE-1 — range: [0, 1]
  • Unigram overlap between the generated abstract and the reference abstract.
• ROUGE-2 — range: [0, 1]
  • Bigram overlap between the generated abstract and the reference abstract.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between the generated abstract and the reference abstract.

Input / output format

Input: Text from the abstract or claims section of a patent application.

Output: Binary label (accepted/rejected) for acceptance prediction; IPC subclass code for classification; generated abstract text for summarization.

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_topk_accuracy(preds, gold, k=5):
    # preds: list of lists of (score, label) sorted descending
    top_k_labels = [label for _, label in preds[:k]]
    return 1.0 if gold in top_k_labels else 0.0

def compute_rouge(ref, hyp):
    # ref: list of reference strings, hyp: list of hypothesis strings
    scores = rouge_score(ref, hyp, rouge_types=['rouge1', 'rouge2', 'rougeL'])
    return scores['rouge1'].fmeasure, scores['rouge2'].fmeasure, scores['rougeL'].fmeasure

Common pitfalls

• Test sets for acceptance prediction are strictly balanced (50/50 accepted vs rejected), so the accuracy baseline is 50%, not random chance over the full filing distribution.
• ROUGE scores are not directly comparable to prior work like BIGPATENT due to different evaluation data and tokenization schemes.
• Models are evaluated only on pre-grant filings (2004–2018), not granted patents, which significantly changes the text distribution and label availability.

Evidence (verbatim from paper)

All the test sets contained equal numbers of accepted and rejected applications, so the baseline accuracy to compare these models against is 50%.

Citation

@misc{suzgun2022hupd,
  title={The Harvard USPTO Patent Dataset: A Large-Scale, Well-Structured, and Multi-Purpose Corpus of Patent Applications},
  author={Suzgun et al. (2022)},
  year={2022},
  note={arXiv:2207.04043}
}

• arXiv: 2207.04043