Hipe 2026 Eval

hipe-2026-eval

CLEF HIPE-2026: Evaluating Accurate and Efficient Person-Place Relation Extraction from Multilingual Historical Texts — Opitz et al. (2026) (arXiv:2602.17663, 2026)

What this evaluates

Evaluates multilingual historical text systems on person-place relation extraction, requiring temporal and geographical reasoning to classify relations as 'at' or 'isAt' with nuanced evidence levels (true, probable, false). It probes both extraction accuracy and reasoning quality in noisy, sparse corpora while also measuring computational efficiency.

Datasets

• HIPE-2026 — total ?; splits: test_a (-1), test_b (-1); repo https://github.com/hipe-eval/HIPE-2026-data

Metrics

• macro-averaged Recall (primary) — range: [0, 1]
  • MacroRecall = (1/|L|) * Σ_{ℓ∈L} Recall(ℓ), where Recall(ℓ) = (#examples with label ℓ correctly predicted) / (#examples whose gold label is ℓ). For Test Set A, macro Recall is computed separately for the 'at' and 'isAt' relations and then averaged.

Input / output format

Input: Multilingual historical text passages containing person and place entities, requiring classification of their spatial-temporal relations.

Output: Classification of each person-place pair into relation type ('at' or 'isAt') and evidence strength ('true', 'probable', or 'false').

Scoring recipe

def compute_macro_recall(predictions, golds):
    labels = set(golds)
    recalls = []
    for label in labels:
        tp = sum(1 for p, g in zip(predictions, golds) if p == label == g)
        fn = sum(1 for g in golds if g == label) - tp
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        recalls.append(recall)
    return sum(recalls) / len(recalls)

Common pitfalls

• Computing macro-recall across all labels jointly instead of separately for 'at' and 'isAt' relations before averaging, as required for Test Set A.
• Using standard accuracy instead of macro-averaged Recall, which fails to account for class imbalance across evidence labels (true/probable/false).
• Assuming efficiency metrics are automatically measured; the protocol relies on self-reported parameter counts and model sizes via survey, making cross-system comparisons non-standardized.

Evidence (verbatim from paper)

System responses are evaluated per test set using macro-averaged Recall (also known as balanced accuracy). Macro Recall is defined as follows: For each label ℓ in the label set L, compute its recall: Recall(ℓ) = #examples with label ℓ correctly predicted / #examples whose gold label is ℓ. The final score is the arithmetic mean of the per-label recalls: MacroRecall = (1/|L|)∑_{ℓ∈L} Recall(ℓ). For Test Set A, which includes three labels for at and two labels for isAt, macro Recall is computed separately for each relation and then averaged to obtain the final system ranking.

Citation

@misc{opitz2026hipe2026,
  title={CLEF HIPE-2026: Evaluating Accurate and Efficient Person-Place Relation Extraction from Multilingual Historical Texts},
  author={Opitz et al. (2026)},
  year={2026},
  note={arXiv:2602.17663}
}

• arXiv: 2602.17663