Nep Glu Eval

nep-glu-eval

Development of Pre-Trained Transformer-based Models for the Nepali Language — Thapa et al. (2024) (arXiv:2411.15734, 2024)

What this evaluates

Evaluates encoder-based language models on core Nepali natural language understanding tasks, including named entity recognition, part-of-speech tagging, text classification, and categorical pair similarity.

Datasets

• Nep-gLUE — total ?; splits: test (-1)

Metrics

• Nep-gLUE Score (primary) — range: percent
  • Macro-average of task-specific accuracy scores across four subtasks: NER, POS tagging, text classification (CC), and categorical pair similarity (CPS).

Input / output format

Input: Tokenized text sequences or sentence pairs for NLU tasks (NER, POS, classification, similarity).

Output: Task-specific predictions: entity labels, POS tags, class labels, or similarity scores.

Scoring recipe

def nep_glu_score(predictions, golds):
    task_scores = []
    for task in ['NER', 'POS', 'CC', 'CPS']:
        task_scores.append(accuracy(predictions[task], golds[task]))
    return sum(task_scores) / len(task_scores)

Common pitfalls

• Hyperparameter tuning is task-specific (3-10 epochs, learning rates from 1e-5 to 5e-5); best model is selected per task on the test set, not via validation.
• The aggregate Nep-gLUE score masks per-task performance; a model can score high overall while underperforming on specific linguistic phenomena.

Evidence (verbatim from paper)

We used the Nepali Language Evaluation Benchmark, or Nep-gLUE Timilsina et al. ([2022]). It consists of four tasks, including Named Entity Recognition (NER), Part-of-Speech (POS) Tagging, text classification, and categorical pair similarity... Our models outperformed all existing models across all tasks, scoring 95.60 on Nep-gLUE Timilsina et al. ([2022]) benchmark

Citation

@misc{thapa2024development,
  title={Development of Pre-Trained Transformer-based Models for the Nepali Language},
  author={Thapa et al. (2024)},
  year={2024},
  note={arXiv:2411.15734}
}

• arXiv: 2411.15734