Nusax Eval

nusax-eval

NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages — Winata et al. (2022) (arXiv:2205.15960, 2022)

What this evaluates

Evaluates sentiment classification and machine translation capabilities across 10 low-resource Indonesian local languages, Indonesian, and English. It probes cross-lingual transferability, multilingual training benefits, and data efficiency for underrepresented Austronesian languages.

Datasets

• NusaX — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/IndoNLP/nusax

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score across all sentiment classes, computed per language and averaged for overall results.
• SacreBLEU — range: [0, 100]
  • Standard BLEU score with sentence-level n-gram matching and brevity penalty, computed using the SacreBLEU toolkit.

Input / output format

Input: For sentiment analysis: a single sentence in one of the 12 languages (10 local Indonesian languages, Indonesian, or English). For machine translation: a source sentence in a source language (e.g., Indonesian, English, or a local language).

Output: For sentiment analysis: a predicted sentiment class label. For machine translation: a translated sentence in the target language.

Scoring recipe

def compute_macro_f1(preds, golds):
    classes = sorted(set(preds + golds))
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)

def compute_sacrebleu(preds, golds):
    return sacrebleu.corpus_bleu(preds, [golds]).score

Common pitfalls

• Copy baseline scores can be artificially high for languages with high lexical or grammatical overlap with Indonesian, misleadingly suggesting good translation performance.
• SacreBLEU may not capture semantic fidelity or handle rare words well, as models often copy or mistranslate them due to limited training data.
• Cross-lingual transfer results can be inflated by shared vocabulary or similar syntax rather than true generalization, especially between closely related Austronesian languages.

Evidence (verbatim from paper)

Table 4: Results of the machine translation task from other languages to Indonesian (x → ind) based on SacreBLEU. Table 7: Sentiment analysis results for macro-F1 (%) of XLM-R_LARGE in the multilingual setting.

Citation

@misc{winata2022nusax,
  title={NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages},
  author={Winata et al. (2022)},
  year={2022},
  note={arXiv:2205.15960}
}

• arXiv: 2205.15960