Indonesian Pos Tagging Eval

indonesian-pos-tagging-eval

Toward a Standardized and More Accurate Indonesian Part-of-Speech Tagging — Kurniawan et al. (2018) (arXiv:1809.03391, 2018)

What this evaluates

Evaluates sequence labeling performance on Indonesian text by assigning part-of-speech tags to tokens. It probes morphological feature extraction, contextual understanding, and robustness to annotation inconsistencies and rare lexical categories.

Datasets

• IDN Tagged Corpus — total ?; splits: 5-fold CV (-1); repo https://github.com/kmkurn/id-pos-tagging

Metrics

• F1 (primary) — range: [0, 1]
  • Token-level F1 score computed over all POS tags. Calculated as 2 * (precision * recall) / (precision + recall), where precision and recall are derived from exact match counts between predicted and gold tags across all tokens.

Input / output format

Input: A sequence of Indonesian words/tokens.

Output: A sequence of POS tags corresponding to each input token.

Scoring recipe

def compute_f1(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    total = len(gold)
    precision = correct / total
    recall = correct / total
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1

Common pitfalls

• Annotation inconsistency for WH and SC tags causes models to struggle with these specific categories.
• The rare 'X' tag contains typos, slang, and foreign terms, often misclassified as nouns due to ambiguous morphological cues.
• Evaluating without 5-fold cross-validation may yield unreliable results due to dataset distribution variance.

Evidence (verbatim from paper)

The F1 scores are averaged over the 5 cross-validation folds. We see that Major baseline performs very poorly compared with the Memo baseline, which surprisingly achieves over 90 F1 points. This result suggests that Memo is a more suitable baseline for this dataset in contrast with Major. The result also provides evidence to the usefulness of our evaluation metric which heavily penalizes a simple majority vote model. Furthermore, we notice that the rule-based tagger by Rashel et al. performs worse than Memo, indicating that Memo is not just suitable but also quite a strong baseline. Moving on, we observe how CRF has 6 points advantage over Memo, signaling that incorporating contextual features and modeling tag-to-tag transitions are useful. Lastly, the biLSTM with CRF tagger performs the best with 97.47 F1 score.

Citation

@misc{kurniawan2018indonesian,
  title={Toward a Standardized and More Accurate Indonesian Part-of-Speech Tagging},
  author={Kurniawan et al. (2018)},
  year={2018},
  note={arXiv:1809.03391}
}

• arXiv: 1809.03391