qhjqhj00 klue-dp-eval
KLUE: Korean Language Understanding Evaluation — Sungjoon Park et al. (arXiv:2105.09680, 2021)
What this evaluates
Evaluates a model's ability to predict syntactic dependency relations between words in Korean sentences, testing grammatical structure understanding.
Datasets
- KLUE-DP — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/KLUE-benchmark/KLUE
Metrics
LAS(primary) — range: [0, 1]- Labeled Attachment Score: percentage of words where both the head word and dependency relation label are predicted correctly.
Input / output format
Input: Korean sentence with tokenized input.
Output: List of (head_index, relation_label) pairs per token.
Scoring recipe
def compute_las(pred_deps, gold_deps):
correct = sum(1 for p, g in zip(pred_deps, gold_deps) if p == g)
return correct / len(gold_deps)
Common pitfalls
- Korean agglutinative morphology complicates token-to-word alignment.
- Head-finding rules differ between UD guidelines and paper-specific annotations.
Evidence (verbatim from paper)
KLUE introduces a comprehensive, ethically designed benchmark for Korean NLU with 8 tasks (Topic Classification, STS, NLI, NER, RE, DP, MRC, DST) built from scratch using diverse, copyright-respected corpora.
Citation
@misc{park2021klue,
title={KLUE: Korean Language Understanding Evaluation},
author={Sungjoon Park et al.},
year={2021},
note={arXiv:2105.09680}
}
- arXiv: 2105.09680