Basqueglue Eval

basqueglue-eval

Latxa: An Open Language Model and Evaluation Suite for Basque — Etxaniz et al. (2024) (arXiv:2403.20266, 2024)

What this evaluates

This benchmark evaluates Basque language models across classical NLP tasks including topic classification, stance detection, coreference detection, and natural language inference. It measures both task-specific performance and overall linguistic competence in a low-resource agglutinative language setting.

Datasets

• BasqueGLUE — total ?; splits: test (-1); repo https://github.com/hitz-zentroa/atxa

Metrics

• F1 — range: [0, 1]
  • Standard F1 score for classification tasks. For VaxxStance, it is computed as a macro-average over the IN FAVOUR and AGAINST categories only.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted labels out of total instances.
• Avg (primary) — range: [0, 1]
  • Arithmetic mean of the task-specific scores (F1 or Accuracy) across all BasqueGLUE tasks.

Input / output format

Input: Text prompts with task instructions (few-shot or zero-shot for decoder models) or raw text sequences (for encoder models).

Output: Predicted class label or probability distribution over classes.

Scoring recipe

def compute_avg_score(task_results):
    scores = []
    for task, res in task_results.items():
        if task == 'VaxxStance':
            scores.append(macro_f1(res['golds'], res['preds'], pos=['IN FAVOUR', 'AGAINST']))
        else:
            scores.append(f1_score(res['golds'], res['preds']) if 'F1' in task else accuracy_score(res['golds'], res['preds']))
    return sum(scores) / len(scores)

Common pitfalls

• VaxxStance F1 is macro-averaged only over IN FAVOUR and AGAINST, excluding other categories.
• Evaluation settings vary by model type (few-shot for decoders vs. fine-tuned for encoders), making direct performance comparisons across architectures tricky.
• The average score aggregates heterogeneous tasks without normalization, which can skew the overall benchmark score.

Evidence (verbatim from paper)

Table 6: BasqueGLUE results by task. *VaxxStance is measured in terms of macro-average F1-score of the categories IN FAVOUR and AGAINST. †BERTeus and ElhBERTeu are fine-tuned encoders.

Citation

@misc{etxaniz2024latxa,
  title={Latxa: An Open Language Model and Evaluation Suite for Basque},
  author={Etxaniz et al. (2024)},
  year={2024},
  note={arXiv:2403.20266}
}

• arXiv: 2403.20266