Halvest Contrastive Eval

halvest-contrastive-eval

Harvesting Textual and Contrastive Data from the HAL Publication Repository — Kulumba et al. (2024) (arXiv:2407.20595, 2024)

What this evaluates

Evaluates language models' ability to capture authorial style and stylometric patterns in scholarly text, independent of topical content. It probes whether models can distinguish documents by the same author across different topics and languages using triplet classification and document retrieval tasks.

Datasets

• HALvest-Contrastive — total ?; splits: base (-1), unrestricted (-1), ICT (-1); repo https://github.com/Madjakul/HALvesting

Metrics

• Accuracy (primary) — range: percent
  • Fraction of correctly classified triplets (query, positive, negative) out of the total number of test triplets, reported as a percentage.
• Recall@10 — range: percent
  • Proportion of queries where the true author's document appears in the top 10 retrieved candidates from a pool of ~14,300, reported as a percentage.
• nDCG@10 — range: percent
  • Normalized Discounted Cumulative Gain at rank 10, measuring the quality of the top-10 ranked list relative to the ideal ranking, reported as a percentage.

Input / output format

Input: Sentence triplets (query, positive, negative) for triplet classification, or a query document paired with a candidate pool of ~14,300 documents for retrieval.

Output: For triplet classification: predicted class label (positive or negative). For retrieval: a ranked list of candidate documents.

Scoring recipe

import math

def score_triplets(preds, golds):
    return sum(1 for p, g in zip(preds, golds) if p == g) / len(golds) * 100

def score_recall_at_10(retrieved_lists, gold_docs):
    recalls = [1.0 if any(g in r[:10] for g in gold_docs[i]) else 0.0 for i, r in enumerate(retrieved_lists)]
    return sum(recalls) / len(recalls) * 100

def score_ndcg_at_10(retrieved, gold):
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(retrieved[:10]) if doc in gold)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(gold), 10)))
    return (dcg / idcg) * 100 if idcg > 0 else 0.0

Common pitfalls

• Unrestricted triplets contain high topical overlap (Jaccard similarity ~0.11 for 8 sentences), allowing models to cheat by matching keywords rather than learning style.
• Sequence length significantly impacts performance; BM25 gains ~20% from 2 to 8 sentences due to lexical signal, while neural models plateau earlier, requiring careful length normalization.
• The retrieval candidate pool is large (14.3k), making random baseline nDCG@10 extremely low (0.07%), so modest absolute scores actually indicate strong stylometric signal.

Evidence (verbatim from paper)

accuracy is reported as the primary metric. It achieves a normalized discounted cumulative gain (nDCG@10) of 19.84%. While modest in isolation, this score is exceptionally strong given the task’s difficulty: a random baseline, which shuffles the 14.3k candidates we had in our test set, would yield an nDCG@10 of $\frac{10}{14300}\approx 0.07%$.

Citation

@misc{kulumba2024halvesting,
  title={Harvesting Textual and Contrastive Data from the HAL Publication Repository},
  author={Kulumba et al. (2024)},
  year={2024},
  note={arXiv:2407.20595}
}

• arXiv: 2407.20595