Indonlu Eval

indonlu-eval

IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding — Wilie et al. (2020) (arXiv:2009.05387, 2020)

What this evaluates

This benchmark evaluates Indonesian natural language understanding across 12 diverse tasks, including single-sentence classification, sentence-pair classification, and sequence labeling/tagging. It probes a model's ability to handle sentiment analysis, aspect-based sentiment, textual entailment, part-of-speech tagging, named entity recognition, keyphrase extraction, and question answering in Indonesian.

Datasets

• IndoNLU — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• macro-averaged F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score computed separately for classification and sequence labeling tasks. For classification, it uses top-1 predictions. For sequence labeling, it uses word-level IOB matching following the CoNLL evaluation script. Two separate mean F1 scores are reported.

Input / output format

Input: Tokenized Indonesian text (single sentence or sentence pair) with corresponding gold labels (class labels for classification, IOB tags for sequence labeling).

Output: Predicted class labels (top-1) for classification tasks, or predicted IOB tag sequences for sequence labeling tasks.

Scoring recipe

def compute_macro_f1(preds, golds, task_type):
    if task_type == 'classification':
        preds = [p.argmax() for p in preds]
    f1_scores = []
    for label in unique_labels:
        tp = sum(1 for p, g in zip(preds, golds) if p == label and g == label)
        fp = sum(1 for p, g in zip(preds, golds) if p == label and g != label)
        fn = sum(1 for p, g in zip(preds, golds) if p != label and g == label)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Using micro-averaged F1 instead of the specified macro-averaged F1.
• Reporting a single overall F1 score instead of two separate mean F1 scores for classification and sequence labeling tasks.
• Evaluating sequence labeling at the character or span level rather than using the specified word-level IOB matching protocol.

Evidence (verbatim from paper)

We use the F1 score to measure the evaluation performance of all tasks. For the binary and multi-label classification tasks, we measure the macro-averaged F1 score by taking the top-1 prediction from the model. For the sequence labeling task, we calculate word-level sequence labeling macro-averaged F1-score for all models by following the sequence labeling evaluation method described in the CoNLL evaluation script. We calculate two mean F1-scores separately for classification and sequence labeling tasks to evaluate models on our IndoNLU benchmark.

Citation

@misc{wilie2020indonlu,
  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
  author={Wilie et al. (2020)},
  year={2020},
  note={arXiv:2009.05387}
}

• arXiv: 2009.05387