South Slavic Encoder Eval

south-slavic-encoder-eval

Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining — Ljubešić et al. (2024) (arXiv:2404.05428, 2024)

What this evaluates

Evaluates multilingual encoder models on South Slavic languages (Croatian and Serbian) across three diverse NLP tasks: named entity recognition, parliamentary sentiment regression, and causal commonsense reasoning. Tests whether cost-efficient additional pretraining can match dedicated monolingual encoders without full from-scratch training.

Datasets

• hr500k — total 499635; splits: train (-1), dev (-1), test (-1)
• ReLDI-NormTagNER-hr — total 89855; splits: train (-1), dev (-1), test (-1)
• SETimes.SR — total 92271; splits: train (-1), dev (-1), test (-1)
• ReLDI-NormTagNER-sr — total 97673; splits: train (-1), dev (-1), test (-1)
• ParlaSent (HBS) — total 5200; splits: train (2600), test (2600)
• COPA (Croatian & Serbian) — total 1000; splits: train (400), dev (100), test (500)

Metrics

• macro F1 score (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across all entity types and datasets, aggregated over 3 random runs.
• R^2 score — range: other
  • Coefficient of determination for ordinal regression (0–5 scale), averaged over 5 random runs.
• accuracy — range: [0, 1]
  • Percentage of correctly predicted plausible alternatives, averaged over 10 random runs.

Input / output format

Input: Tokenized text sequences for NER; single sentences for sentiment; premise sentence paired with two candidate continuations for COPA.

Output: Token-level entity labels for NER; ordinal regression score (0–5) for sentiment; binary class index (0 or 1) selecting the more plausible alternative for COPA.

Scoring recipe

def score_ner(preds_list, gold_list):
    f1s = [macro_f1(p, g) for p, g in zip(preds_list, gold_list)]
    return mean(f1s)

def score_sentiment(preds, gold):
    return r_squared(preds, gold)

def score_copa(preds, gold):
    return accuracy(preds, gold)

# Aggregation over runs
ner_final = mean([score_ner(p, g) for p, g in run_3_times()])
sent_final = mean([score_sentiment(p, g) for p, g in run_5_times()])
copa_final = mean([score_copa(p, g) for p, g in run_10_times()])

Common pitfalls

• Hyperparameter tuning is task- and dataset-specific: separate searches are performed for standard vs. non-standard NER datasets due to size/diversity differences, and sentiment uses a marked validation subset rather than a held-out dev set.
• Sentiment is treated as a regression task with an ordinal 0–5 scale, not a classification task, so accuracy is not reported for it.
• Results are not single-run values; they must be averaged over multiple random seeds (3 for NER, 5 for sentiment, 10 for COPA) to match the paper's reporting protocol.

Evidence (verbatim from paper)

For named entity recognition, we train and test each model three times and report aggregated results in the macro F1 score. For sentiment, we perform five runs, and report average $R^{2}$ scores. For COPA, we average over 10 runs and report the accuracy score.

Citation

@misc{ljubesic2024diet,
  title={Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining},
  author={Ljubešić et al. (2024)},
  year={2024},
  note={arXiv:2404.05428}
}

• arXiv: 2404.05428