Bigpatent Eval

bigpatent-eval

BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization — Sharma et al. (2019) (arXiv:1906.03741, 2019)

What this evaluates

Evaluates abstractive summarization models on patent documents, probing their ability to capture global discourse structure, maintain entity coherence, and generate novel content without excessive repetition or fabrication.

Datasets

• BIGPATENT — total ?; splits: test (-1)

Metrics

• ROUGE-1 F1 (primary) — range: percent
  • F1 score computed between the generated summary and the human-written reference summary using unigram overlap.
• ROUGE-2 F1 — range: percent
  • F1 score computed between the generated summary and the human-written reference summary using bigram overlap.
• ROUGE-L F1 — range: percent
  • F1 score based on the longest common subsequence between the generated summary and the reference summary.

Input / output format

Input: Patent document text (input articles).

Output: Abstractive summary text (truncated to 100 words/tokens during evaluation).

Scoring recipe

def compute_rouge_f1(generated, reference):
    gen_tokens = tokenize(generated)
    ref_tokens = tokenize(reference)
    gen_ngrams = ngrams(gen_tokens, n=1)
    ref_ngrams = ngrams(ref_tokens, n=1)
    overlap = sum(min(gen_ngrams[g], ref_ngrams[g]) for g in set(gen_ngrams) & set(ref_ngrams))
    precision = overlap / len(gen_ngrams) if len(gen_ngrams) > 0 else 0
    recall = overlap / len(ref_ngrams) if len(ref_ngrams) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100

Common pitfalls

• Models are evaluated with inputs and summaries truncated to 400 and 100 respectively, which may disadvantage models designed for long documents.
• ROUGE scores do not capture entity repetition or fabrication, which are significant failure modes in patent summarization.
• Extractive baselines like LEAD-3 perform poorly due to the uniform distribution of salient content in patents, unlike news articles.

Evidence (verbatim from paper)

Table 4 reports F1 scores of ROUGE-1, 2, and L (Lin and Hovy, 2003) for all models.

Citation

@misc{sharma2019bigpatent,
  title={BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization},
  author={Sharma et al. (2019)},
  year={2019},
  note={arXiv:1906.03741}
}

• arXiv: 1906.03741