Ltzglue Eval

ltzglue-eval

ltzGLUE: Luxembourgish General Language Understanding Evaluation — Plum et al. (2026) (arXiv:2604.17976, 2026)

What this evaluates

Evaluates Luxembourgish (LTZ) language understanding across eight diverse NLU tasks, including classification, sequence labeling, and textual entailment. It probes encoder models and prompted LLMs on their ability to handle low-resource language nuances, structural complexity, and label sensitivity.

Datasets

• ltzGLUE — total ?; splits: test (-1); repo https://github.com/plumaj/ltzGLUE

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Unweighted mean of per-class F1 scores. Computed as 2 * (precision * recall) / (precision + recall) for each class, then averaged across all classes.

Input / output format

Input: Luxembourgish text inputs for classification, sequence labeling, or entailment tasks.

Output: Predicted class labels or token-level tags corresponding to the specific task.

Scoring recipe

def compute_macro_f1(predictions, gold):
    classes = sorted(set(predictions) | set(gold))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Prompted LLMs may produce malformed outputs or incorrect numbers of predictions, which are discarded prior to evaluation.
• Encoder results are averaged over three runs with standard deviations reported, while LLMs are evaluated only once, making stability comparisons difficult.
• High class imbalance in HA and SA requires class-balanced loss during fine-tuning, which affects training but not the test metric itself.

Evidence (verbatim from paper)

Table 6 shows F1 scores for all models across all tasks (see Appendix 7.9 for full results). Prompted LLMs were evaluated once; we report macro-F1 only.

Citation

@misc{plum2026ltzglue,
  title={ltzGLUE: Luxembourgish General Language Understanding Evaluation},
  author={Plum et al. (2026)},
  year={2026},
  note={arXiv:2604.17976}
}

• arXiv: 2604.17976