Patent Ce Eval

patent-ce-eval

Towards Better Evaluation for Generated Patent Claims — Jiang et al. (2025) (arXiv:2505.11095, 2025)

What this evaluates

This benchmark evaluates the quality of generated patent claims against expert-annotated reference claims across five dimensions: feature completeness, conceptual clarity, terminology consistency, logical linkage, and overall quality. It probes a model's ability to capture patent-specific linguistic precision, legal formality, and structural requirements rather than just surface-level text overlap.

Datasets

• Patent-CE — total 1228; splits: test (-1); repo https://github.com/scylj1/PatClaimEval

Metrics

• Kendall-Tau correlation (primary) — range: [-1, 1]
  • Measures the number of concordant and discordant pairs between two ranked lists to assess rank correlation. Values range from -1 (perfect disagreement) to 1 (perfect agreement).
• Spearman correlation — range: [-1, 1]
  • Computes the Pearson correlation between the rank variables of two lists. It is more sensitive to large rank differences than Kendall-Tau.
• accuracy — range: [0, 1]
  • Proportion of correctly predicted classification labels in the three-way classification setup.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for the three-way classification task.

Input / output format

Input: A pair of texts: a reference patent claim and a candidate (generated) patent claim.

Output: A quality score or rank for the candidate claim, or a three-way classification label (e.g., acceptable/unacceptable/conditional), depending on the metric being evaluated.

Scoring recipe

def compute_kendall_tau(predictions, gold):
    concordant = discordant = tied = 0
    for i in range(len(predictions)):
        for j in range(i + 1, len(predictions)):
            diff_p = predictions[i] - predictions[j]
            diff_g = gold[i] - gold[j]
            if diff_p * diff_g > 0: concordant += 1
            elif diff_p * diff_g < 0: discordant += 1
            else: tied += 1
    tau = (concordant - discordant) / (concordant + discordant + tied)
    return tau

Common pitfalls

• The dataset explicitly lacks source texts/context, so metrics requiring context (like AlignScore or UniEval) cannot be applied as intended.
• For classification tasks, score differences below 10^-4 are treated as equivalent labels, which can artificially inflate accuracy/F1 if not handled.
• Evaluating patent claims requires multi-dimensional assessment (completeness, clarity, consistency, linkage, quality) rather than relying solely on n-gram overlap or single scalar scores.

Evidence (verbatim from paper)

We used the Kendall-Tau correlation to assess the overall alignment with human judgment, following the approach of previous work by Zuo et al. ([2024]). This correlation metric evaluates the consistency of the global ranking while disregarding minor errors in individual predictions. We additionally report the Spearman correlation. Compared to Kendall-Tau, Spearman is more sensitive to large rank differences, providing a complementary perspective on the metric ability to predict relative claim quality.

Citation

@misc{jiang2025patentce,
  title={Towards Better Evaluation for Generated Patent Claims},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2505.11095}
}

• arXiv: 2505.11095