Kazsandra Eval

kazsandra-eval

KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes — Yeshpanov et al. (2024) (arXiv:2403.19335, 2024)

What this evaluates

Evaluates multilingual sentiment classification models on Kazakh customer reviews, probing their ability to handle code-switching, mixed scripts, and imbalanced class distributions across polarity and numerical score prediction tasks.

Datasets

• KazSAnDRA — total 180064; splits: train (-1), val (-1), test (-1); repo https://github.com/IS2AI/KazSAnDRA

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Macro-averaged F1-score, calculated as the unweighted arithmetic mean of the F1-scores for each class. F1 for a class is 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Raw text customer reviews in Kazakh, often containing code-switching and mixed scripts (Cyrillic/Latin/Russian).

Output: For Polarity Classification (PC): binary sentiment label (positive/negative). For Score Classification (SC): integer rating from 1 to 5.

Scoring recipe

def macro_f1(predictions, gold):
    classes = sorted(set(predictions) | set(gold))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Using micro-averaging instead of macro-averaging, which would mask poor performance on minority classes in the imbalanced dataset.
• Assuming score classification (SC) will perform similarly to polarity classification (PC); SC is significantly harder (F1 ~0.39 vs 0.81) due to fine-grained rating prediction.
• Ignoring code-switching and mixed-script variations in Kazakh reviews, which can degrade model performance if not handled during preprocessing or fine-tuning.

Evidence (verbatim from paper)

Several conventional metrics were used to evaluate the performance of the models, including accuracy (A), precision (P), recall (R), and F${}{1}$-score (F${}{1}$). Given the imbalanced nature of the dataset, where all classes carry equal importance, we opted for macro-averaging, calculated from the arithmetic (i.e., unweighted) mean of all F${}_{1}$-scores per class, and thus ensuring equal treatment of all classes during the evaluation, resulting in a stronger penalty if the model performs worse on minority classes

Citation

@misc{yeshpanov2024kazsandra,
  title={KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes},
  author={Yeshpanov et al. (2024)},
  year={2024},
  note={arXiv:2403.19335}
}

• arXiv: 2403.19335