Latvian Encoder Eval

latvian-encoder-eval

Pretraining and Benchmarking Modern Encoders for Latvian — Znotins (2026) (arXiv:2603.15005, 2026)

What this evaluates

Evaluates Latvian-specific encoder models on lightweight diagnostic tasks, morphosyntactic parsing, and semantic representation quality to benchmark low-resource language modeling capabilities.

Datasets

• EuroEval Latvian diagnostics — total ?; splits: train (1024), val (256), test (2048)
• COPA (Latvian) — total ?; splits: train (400), val (100), test (500); HF AiLab-IMCS-UL/copa-lv
• Universal Dependencies Latvian treebank (UD v2.16) — total 19000; splits: train (-1), val (-1), test (-1)
• Latvian WSD dataset — total 54364; splits: train (-1), val (200), test (500); HF AiLab-IMCS-UL/wsd-lv

Metrics

• MCC (primary) — range: [-1, 1]
  • Matthews correlation coefficient for binary/multiclass classification.
• macro-F1 (MF1) — range: [0, 1]
  • Macro-averaged F1 score across all classes.
• UAS / LAS — range: percent
  • Unlabeled/Labelled Attachment Score for dependency parsing, computed via official CoNLL script.
• top-1 sense selection accuracy — range: [0, 1]
  • Accuracy of selecting the correct gloss/sense from the full primary sense inventory.

Input / output format

Input: Tokenized text sequences (sentences, passages, or context-target pairs) fed into encoder models for fine-tuning.

Output: Task-specific predictions: class labels, dependency arcs/relations, or candidate gloss IDs.

Scoring recipe

def score(preds, golds, task):
    if task == 'classification':
        return {'MCC': matthews_corrcoef(golds, preds), 'MF1': f1_score(golds, preds, average='macro')}
    elif task == 'parsing':
        return conll_eval_script(golds, preds) # UAS, LAS, MLAS, BLEX
    elif task == 'wsd':
        return {'top1_acc': accuracy_score(golds, preds)}

Common pitfalls

• Original EuroEval hyperparameters (lr=2e-5, patience=2) are suboptimal for Latvian; must use searched lr and patience=5.
• WSD dataset splits by lemma entry to strictly prevent lexical overlap between training and evaluation sets.
• UD parsing uses a biaffine graph-based parser with maximum spanning tree decoding under a single-root constraint, not standard beam search.

Evidence (verbatim from paper)

We evaluate performance using Matthews correlation coefficient (MCC) and macro-F1 (MF1). ... We report standard UD metrics computed with the official CoNLL UD evaluation script, including token-level tagging accuracy for UPOS, XPOS, UFeats, AllTags, and Lemmas, as well as dependency parsing quality measured by UAS, LAS, CLAS, MLAS, and BLEX. ... We evaluate performance using binary accuracy on context–sense pairs and top-1 sense selection accuracy over the full primary sense inventory.

Citation

@misc{znotins2026pretraining,
  title={Pretraining and Benchmarking Modern Encoders for Latvian},
  author={Znotins (2026)},
  year={2026},
  note={arXiv:2603.15005}
}

• arXiv: 2603.15005