Kyrgyz Sst2 Eval

kyrgyz-sst2-eval

KyrgyzBERT: A Compact, Efficient Language Model for Kyrgyz NLP — Metinov et al. (2025) (arXiv:2511.20182, 2025)

What this evaluates

Evaluates sentiment classification capability on Kyrgyz language text. It measures how effectively a model can distinguish between positive and negative sentiments using a manually annotated benchmark dataset.

Datasets

• kyrgyz-sst2 — total ?; splits: test (-1)

Metrics

• F1-score (Weighted) (primary) — range: [0, 1]
  • Weighted F1-score, which computes the F1-score for each sentiment class and averages them weighted by the number of true instances for each class.

Input / output format

Input: Kyrgyz language text sentences.

Output: Predicted sentiment label (positive or negative) per sentence.

Scoring recipe

def weighted_f1(preds, gold):
    classes = sorted(set(preds) | set(gold))
    f1s, weights = [], []
    for c in classes:
        tp = sum(p == c and g == c for p, g in zip(preds, gold))
        fp = sum(p == c and g != c for p, g in zip(preds, gold))
        fn = sum(p != c and g == c for p, g in zip(preds, gold))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
        weights.append(sum(1 for g in gold if g == c))
    return sum(f * w for f, w in zip(f1s, weights)) / sum(weights)

Common pitfalls

• Zero-shot base models perform near random chance (~0.32 F1), so readers must not assume pre-trained multilingual models work out-of-the-box for Kyrgyz.
• The benchmark reports weighted F1, not accuracy; comparing against accuracy-based baselines or misinterpreting the metric will lead to incorrect conclusions about model performance.

Evidence (verbatim from paper)

The results clearly indicate that task-specific finetuning is essential, as the base models perform poorly, with F1-scores near the level of a random classifier (Table [II]). Our primary finding is that finetuned ‘KyrgyzBert‘ achieves a strong F1-score of 0.8280, confirming that a compact, monolingual model is highly effective for downstream tasks in Kyrgyz.

Citation

@misc{metinov2025kyrgyzbert,
  title={KyrgyzBERT: A Compact, Efficient Language Model for Kyrgyz NLP},
  author={Metinov et al. (2025)},
  year={2025},
  note={arXiv:2511.20182}
}

• arXiv: 2511.20182