Hansel Eval

hansel-eval

Hansel: A Chinese Few-Shot and Zero-Shot Entity Linking Benchmark — Xu et al. (2022) (arXiv:2207.13005, 2022)

What this evaluates

Evaluates Chinese entity linking models on few-shot and zero-shot scenarios, specifically probing their ability to link mentions to tail and emerging Wikidata entities without relying on head entity popularity or dataset-specific fine-tuning.

Datasets

• Hansel — total ?; splits: Hansel-FS (-1), Hansel-ZS (-1); repo https://github.com/HITsz-TMG/Hansel
• TAC-KBP2015 — total ?; splits: train (-1), test (-1)

Metrics

• R@1 (primary) — range: percent
  • Recall@K measures the fraction of test instances where the ground-truth entity ID appears within the top-K predicted candidates. R@1 is equivalent to exact-match accuracy.

Input / output format

Input: Mention text with surrounding context, evaluated against a target knowledge base (Wikidata) containing entity descriptions, aliases, and coarse types.

Output: A ranked list of candidate entity QIDs (or a single predicted QID) from the knowledge base.

Scoring recipe

def recall_at_k(pred_candidates, gold_qids, k):
    correct = 0
    for preds, gold in zip(pred_candidates, gold_qids):
        if gold in preds[:k]:
            correct += 1
    return correct / len(gold_qids)

Common pitfalls

• Applying dataset-specific fine-tuning on Hansel, which explicitly violates the zero-shot/few-shot evaluation protocol.
• Ignoring NIL (Not In List) entities or failing to correctly classify them as NIL plus coarse type, leading to inflated scores.
• Using outdated knowledge base dumps that overlap with the test set's emerging entities, causing data leakage.

Evidence (verbatim from paper)

The evaluation metric is Recall@K, where R@1 is equivalent to accuracy. ... When evaluating on Hansel, we do not use dataset-specific tuning. We use AT-base as the alias table and evaluate DE and CA based on AT-base's top-10 candidates.

Citation

@misc{xu2022hansel,
  title={Hansel: A Chinese Few-Shot and Zero-Shot Entity Linking Benchmark},
  author={Xu et al. (2022)},
  year={2022},
  note={arXiv:2207.13005}
}

• arXiv: 2207.13005