Lrebenc Eval

lrebenc-eval

Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study — Xin Xu et al. (2022) (arXiv:2210.10678, 2022)

What this evaluates

Evaluates relation extraction models under low-resource conditions (8-shot, 10%, 100% training data) across diverse domains and languages. It probes few-shot learning capabilities, robustness to long-tailed class distributions, and the effectiveness of data augmentation and self-training strategies.

Datasets

• SemEval 2010 Task 8 — total ?; splits: train (-1), test (-1); repo https://github.com/zjunlp/KnowPrompt/tree/master/dataset/semeval
• TACREV — total ?; splits: train (-1), test (-1); repo https://github.com/DFKI-NLP/tacrev
• DialogRE — total ?; splits: train (-1), test (-1); repo https://dataset.org/dialogre/
• DuIE2.0 — total ?; splits: train (-1), test (-1); repo https://www.luge.ai/#/luge/dataDetail?id=5
• Wiki80 — total ?; splits: train (-1), test (-1); repo https://github.com/thunlp/OpenNRE/blob/master/benchmark/download_wiki80.sh
• ChemProt — total ?; splits: train (-1), test (-1); repo https://github.com/ncbi-nlp/BLUE_Benchmark
• SciERC — total ?; splits: train (-1), test (-1); repo http://nlp.cs.washington.edu/sciIE/
• CMeIE — total ?; splits: train (-1), test (-1); repo https://tianchi.aliyun.com/dataset/dataDetail?dataId=95414

Metrics

• Macro F1 (primary) — range: percent
  • Computes the F1 score for each relation class independently and averages them equally across all classes. Emphasizes performance on minority/long-tail classes.
• Micro F1 — range: percent
  • Aggregates true positives, false positives, and false negatives across all classes globally before computing precision, recall, and F1. Reflects overall instance-level accuracy.

Input / output format

Input: A unified JSON object per instance containing a text string and a relational triple (head entity, tail entity, and target relation label). Multi-triple instances are split into single-triple instances.

Output: A single predicted relation label corresponding to the entity pair in the text.

Scoring recipe

def compute_f1(preds, golds, mode='macro'):
    classes = set(golds)
    if mode == 'macro':
        f1s = []
        for c in classes:
            tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
            fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
            fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0
            f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
        return sum(f1s) / len(f1s)
    else:
        tp = sum(1 for p, g in zip(preds, golds) if p == g)
        fp = sum(1 for p, g in zip(preds, golds) if p != g)
        fn = len(golds) - tp
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Performance is highly sensitive to random data splits in few-shot settings; the paper averages results over 5 random samples to mitigate variance.
• Datasets with multiple triples per instance are split into single-triple instances for evaluation, which alters the original instance distribution and requires careful preprocessing.
• No validation set is used during training to enforce true few-shot evaluation, which can cause training instability and requires strict adherence to the fixed 10-epoch training protocol.

Evidence (verbatim from paper)

Since the performance of head and tail classes varies a lot, we use both Macro F1 and Micro F1 together as the evaluation metrics. We conduct experiments in three settings with different proportions of training data to simulate different resource levels: 8-shot, 10% and 100%.

Citation

@misc{xu2022lrebenc,
  title={Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study},
  author={Xin Xu et al. (2022)},
  year={2022},
  note={arXiv:2210.10678}
}

• arXiv: 2210.10678