Trglue Eval

trglue-eval

Introducing TrGLUE and SentiTurca: A Comprehensive Benchmark for Turkish General Language Understanding and Sentiment Analysis — Altinok (2025) (arXiv:2512.22100, 2025)

What this evaluates

This benchmark evaluates Turkish natural language understanding across multiple task types, including grammaticality judgment, sentiment analysis, paraphrase detection, and semantic textual similarity. It probes a model's ability to handle agglutinative morphology, culturally adapted expressions, and nuanced semantic equivalence in Turkish.

Datasets

• TrCoLA — total 9916; splits: train (-1), dev (-1), test (-1); HF turkish-nlp-suite/TrCoLA; repo https://github.com/turkish-nlp-suite/TrGLUE
• TrSST-2 — total 78000; splits: train (67000), val (8900), test (8900); HF turkish-nlp-suite/BuyukSinema; repo https://github.com/turkish-nlp-suite/TrGLUE
• TrMRPC — total 5180; splits: test (-1); repo https://github.com/turkish-nlp-suite/TrGLUE
• TrSTS-B — total 3060; splits: dev (-1); repo https://github.com/turkish-nlp-suite/TrGLUE

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted labels out of the total number of instances. Standard for classification tasks in the benchmark.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall. Used alongside accuracy to address class imbalance in sentiment analysis tasks.
• Spearman correlation — range: [-1, 1]
  • Rank-based correlation coefficient used to evaluate the monotonic relationship between predicted and gold similarity scores on TrSTS-B.

Input / output format

Input: Single sentence for TrCoLA and TrSST-2; sentence pair for TrMRPC, TrSTS-B, and TrQQP.

Output: Binary label (0 or 1) for TrCoLA, TrSST-2, TrMRPC, and TrQQP; continuous similarity score (1–5) for TrSTS-B.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• TrMRPC uses a relaxed paraphrase definition where sentences sharing the same main claim are labeled equivalent even if one contains extra non-contradictory details, diverging from strict English MRPC rules.
• TrSST-2 exhibits significant class imbalance in its original star ratings, requiring evaluation with both accuracy and F1-score rather than accuracy alone.
• TrSTS-B evaluation is reported exclusively on the development split because the test set was withheld due to the dataset's small size (3.06K pairs).

Evidence (verbatim from paper)

To address this class imbalance, we utilize the both binary accuracy and F1-score, whereas the original GLUE task employs binary accuracy.

Citation

@misc{altinok2025trglue,
  title={Introducing TrGLUE and SentiTurca: A Comprehensive Benchmark for Turkish General Language Understanding and Sentiment Analysis},
  author={Altinok (2025)},
  year={2025},
  note={arXiv:2512.22100}
}

• arXiv: 2512.22100