Klue Tc Eval

klue-tc-eval

KLUE: Korean Language Understanding Evaluation — Sungjoon Park et al. (arXiv:2105.09680, 2021)

What this evaluates

Evaluates a model's ability to classify Korean text into predefined topic categories, testing core semantic understanding and categorization capabilities in Korean.

Datasets

• KLUE-TC — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/KLUE-benchmark/KLUE

Metrics

• Accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted topic labels out of the total number of instances.

Input / output format

Input: Korean text document or sentence.

Output: Predicted topic label.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Data leakage from pretraining corpora if test data overlaps with training sources.
• PII removal may alter text distribution and impact model performance.

Evidence (verbatim from paper)

KLUE introduces a comprehensive, ethically designed benchmark for Korean NLU with 8 tasks (Topic Classification, STS, NLI, NER, RE, DP, MRC, DST) built from scratch using diverse, copyright-respected corpora.

Citation

@misc{park2021klue,
  title={KLUE: Korean Language Understanding Evaluation},
  author={Sungjoon Park et al.},
  year={2021},
  note={arXiv:2105.09680}
}

• arXiv: 2105.09680