qhjqhj00 klue-re-eval
KLUE: Korean Language Understanding Evaluation — Sungjoon Park et al. (arXiv:2105.09680, 2021)
What this evaluates
Tests a model's ability to extract relational triples between entities in Korean text, probing structured information extraction capabilities.
Datasets
- KLUE-RE — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/KLUE-benchmark/KLUE
Metrics
F1(primary) — range: [0, 1]- Harmonic mean of precision and recall for exact triple (head, relation, tail) matching.
Input / output format
Input: Korean sentence with annotated entity spans.
Output: List of predicted relation triples.
Scoring recipe
def compute_f1(pred_triples, gold_triples):
tp = len(set(pred_triples) & set(gold_triples))
fp = len(set(pred_triples) - set(gold_triples))
fn = len(set(gold_triples) - set(pred_triples))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
Common pitfalls
- Overlapping entities require careful disambiguation.
- Relation label granularity varies across Korean corpora.
Evidence (verbatim from paper)
KLUE introduces a comprehensive, ethically designed benchmark for Korean NLU with 8 tasks (Topic Classification, STS, NLI, NER, RE, DP, MRC, DST) built from scratch using diverse, copyright-respected corpora.
Citation
@misc{park2021klue,
title={KLUE: Korean Language Understanding Evaluation},
author={Sungjoon Park et al.},
year={2021},
note={arXiv:2105.09680}
}
- arXiv: 2105.09680