Indolem Eval

indolem-eval

IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP — Koto et al. (2020) (arXiv:2011.00677, 2020)

What this evaluates

Evaluates Indonesian NLP capabilities across morpho-syntax, semantics, and discourse. It probes token-level labeling (POS, NER), syntactic structure (dependency parsing), text classification (sentiment), generation (summarization), and discourse coherence (next tweet prediction, tweet ordering).

Datasets

• INDOLEM — total ?; splits: train (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or classes out of the total number of instances.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall. For NER, computed at the entity level; for classification tasks, typically macro or micro averaged.
• UAS/LAS — range: [0, 1]
  • Unlabeled/Labelled Attachment Score: percentage of words whose head word and dependency label are predicted correctly.
• ROUGE-1/2/L — range: [0, 1]
  • Recall-oriented overlap metrics measuring the similarity between reference and generated summaries using unigrams, bigrams, and longest common subsequence.
• Spearman's ρ — range: [-1, 1]
  • Rank correlation coefficient measuring the monotonic relationship between predicted and true tweet orderings.

Input / output format

Input: Indonesian text inputs including sentences for POS/NER/parsing, tweets for discourse tasks, and source documents for summarization.

Output: Task-specific outputs: token-level labels, entity spans, dependency head/label pairs, sentiment class labels, generated summary text, or ranked/predicted next tweet indices.

Scoring recipe

def compute_metrics(predictions, gold):
    # Accuracy
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    # F1 (entity-level for NER)
    tp = sum(1 for p, g in zip(predictions, gold) if p == g and g != 'O')
    prec = tp / max(1, sum(1 for p in predictions if p != 'O'))
    rec = tp / max(1, sum(1 for g in gold if g != 'O'))
    f1 = 2 * prec * rec / max(1e-9, prec + rec)
    return {'accuracy': acc, 'f1': f1}

Common pitfalls

• Comparing results across studies without verifying test partition consistency (e.g., UD-Indo-PUD versions differ between papers, making direct comparison invalid).
• Human baseline performance was only evaluated on a random 100-instance subset, not the full test set, which may not represent overall human capability on the benchmark.
• POS tagging evaluation uses 5-fold cross-validation, while NER and other tasks report single test-set scores, requiring careful aggregation when comparing model performance.

Evidence (verbatim from paper)

Table 3: Results on POS and NER tasks using accuracy averaged over five folds for POS tagging task, and entity-level F1 over the test set for the NER tasks.

Citation

@misc{koto2020indolem,
  title={IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP},
  author={Koto et al. (2020)},
  year={2020},
  note={arXiv:2011.00677}
}

• arXiv: 2011.00677