# Lrebenc Eval

> Evaluates relation extraction models under low-resource conditions (8-shot, 10%, 100% training data) across diverse domains and languages. It probes few-shot learning capabilities, robustness to long-tailed class distributions, and the effectiveness of data augmentation and self-training strategies. Use when the user wants to benchmark on SemEval 2010 Task 8, TACREV, DialogRE, DuIE2.0, Wiki80, ChemProt, SciERC, CMeIE, or asks about evaluating this task. Reports Macro F1.

- Skill: `qhjqhj00/lrebenc-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/lrebenc-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/lrebenc-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/lrebenc-eval

---


# lrebenc-eval

> Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study — Xin Xu et al. (2022) (arXiv:2210.10678, 2022)

## What this evaluates

Evaluates relation extraction models under low-resource conditions (8-shot, 10%, 100% training data) across diverse domains and languages. It probes few-shot learning capabilities, robustness to long-tailed class distributions, and the effectiveness of data augmentation and self-training strategies.

## Datasets

- **SemEval 2010 Task 8** — total ?; splits: train (-1), test (-1); repo https://github.com/zjunlp/KnowPrompt/tree/master/dataset/semeval
- **TACREV** — total ?; splits: train (-1), test (-1); repo https://github.com/DFKI-NLP/tacrev
- **DialogRE** — total ?; splits: train (-1), test (-1); repo https://dataset.org/dialogre/
- **DuIE2.0** — total ?; splits: train (-1), test (-1); repo https://www.luge.ai/#/luge/dataDetail?id=5
- **Wiki80** — total ?; splits: train (-1), test (-1); repo https://github.com/thunlp/OpenNRE/blob/master/benchmark/download_wiki80.sh
- **ChemProt** — total ?; splits: train (-1), test (-1); repo https://github.com/ncbi-nlp/BLUE_Benchmark
- **SciERC** — total ?; splits: train (-1), test (-1); repo http://nlp.cs.washington.edu/sciIE/
- **CMeIE** — total ?; splits: train (-1), test (-1); repo https://tianchi.aliyun.com/dataset/dataDetail?dataId=95414

## Metrics

- `Macro F1` **(primary)** — range: percent
  - Computes the F1 score for each relation class independently and averages them equally across all classes. Emphasizes performance on minority/long-tail classes.
- `Micro F1` — range: percent
  - Aggregates true positives, false positives, and false negatives across all classes globally before computing precision, recall, and F1. Reflects overall instance-level accuracy.

## Input / output format

**Input**: A unified JSON object per instance containing a text string and a relational triple (head entity, tail entity, and target relation label). Multi-triple instances are split into single-triple instances.

**Output**: A single predicted relation label corresponding to the entity pair in the text.

## Scoring recipe

```python
def compute_f1(preds, golds, mode='macro'):
    classes = set(golds)
    if mode == 'macro':
        f1s = []
        for c in classes:
            tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
            fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
            fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0
            f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
        return sum(f1s) / len(f1s)
    else:
        tp = sum(1 for p, g in zip(preds, golds) if p == g)
        fp = sum(1 for p, g in zip(preds, golds) if p != g)
        fn = len(golds) - tp
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Performance is highly sensitive to random data splits in few-shot settings; the paper averages results over 5 random samples to mitigate variance.
- Datasets with multiple triples per instance are split into single-triple instances for evaluation, which alters the original instance distribution and requires careful preprocessing.
- No validation set is used during training to enforce true few-shot evaluation, which can cause training instability and requires strict adherence to the fixed 10-epoch training protocol.

## Evidence (verbatim from paper)

> Since the performance of head and tail classes varies a lot, we use both Macro F1 and Micro F1 together as the evaluation metrics. We conduct experiments in three settings with different proportions of training data to simulate different resource levels: 8-shot, 10% and 100%.

## Citation

```bibtex
@misc{xu2022lrebenc,
  title={Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study},
  author={Xin Xu et al. (2022)},
  year={2022},
  note={arXiv:2210.10678}
}
```

- arXiv: 2210.10678