Turkish Lm Eval

turkish-lm-eval

Mukayese: Turkish NLP Strikes Back — Safaya et al. (2022) (arXiv:2203.01215, 2022)

What this evaluates

Evaluates a model's ability to capture statistical patterns and long-range dependencies in Turkish text. It tests generative probability estimation at both subword and character levels across news and Wikipedia domains.

Datasets

• trwiki-67 — total 67000000; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese
• trnews-64 — total 64000000; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese

Metrics

• Perplexity (Ppl) (primary) — range: [0, inf)
  • Derivative of negative log-likelihood: Ppl = exp(-1/n * sum(log p_theta(x_i|x_<i))). Evaluated on trwiki-67.
• Bits-per-character (Bpc) (primary) — range: [0, inf)
  • Entropy-based derivative of Nll used for character-level modeling on trnews-64. Lower is better.

Input / output format

Input: Raw cased text (trwiki-67) or lightly pre-processed character-level text (trnews-64).

Output: Probability distribution over the next token or character in the sequence.

Scoring recipe

def score_lm(predictions, gold, dataset):
    n = len(gold)
    nll = -1/n * sum(log(p_theta(x_i | x_<i)) for i in range(n))
    if dataset == 'trwiki-67':
        return exp(nll)  # Ppl
    else:
        return nll / log2(2)  # Bpc (entropy-based derivative)

Common pitfalls

• Ppl must be computed with the same token count or normalized, otherwise results are incomparable.
• Models must only be trained on the provided training split; using external data violates the benchmark protocol.
• Character-level vs subword-level tokenization changes vocabulary size and parameter counts, affecting direct model comparison.

Evidence (verbatim from paper)

Language models are trained on minimizing the negative log-likelihood (Nll) of the training set, and their performance is measured based on how well they can generalize on the test set: Nll(X_test) = -1/n sum log p_theta(x_i|x_test<i). Word or sub-word level language models are evaluated using the word perplexity (Ppl) metric, a derivative of Nll. On the other hand, character language models are evaluated using entropy-based Bits-per-character (Bpc) metric... Note that Ppl needs to be computed with the same count of tokens, otherwise it needs to be normalized

Citation

@misc{safaya2022mukayese,
  title={Mukayese: Turkish NLP Strikes Back},
  author={Safaya et al. (2022)},
  year={2022},
  note={arXiv:2203.01215}
}

• arXiv: 2203.01215