Slovene Superglue Eval

slovene-superglue-eval

Slovene SuperGLUE Benchmark: Translation and Evaluation — Žagar et al. (2022) (arXiv:2202.04994, 2022)

What this evaluates

Evaluates monolingual, cross-lingual, and multilingual NLP models on a human- and machine-translated Slovene version of the SuperGLUE benchmark. It probes how well models handle morphological and grammatical challenges in low-resource language processing, and compares translation quality impacts on downstream task performance.

Datasets

• Slovene SuperGLUE — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Avg (primary) — range: percent
  • Average of task-specific scores across the evaluated tasks. Task scores are Accuracy (Acc.), F1/Accuracy (for CB), or F1a/EM (for MultiRC). Note that the set of tasks averaged differs by experiment (6 tasks in Table 2, 5 tasks excluding WSC in Table 5).
• Accuracy (Acc.) — range: percent
  • Percentage of correctly predicted class labels for tasks like BoolQ, COPA, RTE, and WSC.
• F1/Accuracy — range: percent
  • Reported as F1/Accuracy for the CB task, evaluating both macro-F1 and accuracy.
• F1a/EM — range: percent
  • Reported for MultiRC, evaluating average F1 and exact match for multiple-choice answers.

Input / output format

Input: Slovene text instances corresponding to SuperGLUE tasks (BoolQ, CB, COPA, MultiRC, RTE, WSC), including premises, hypotheses, questions, and context as appropriate for each task.

Output: Predicted class labels or answer spans for each instance.

Scoring recipe

def compute_avg(predictions, gold):
    task_scores = {}
    for task in ['BoolQ', 'CB', 'COPA', 'MultiRC', 'RTE', 'WSC']:
        if task == 'CB':
            task_scores[task] = f1_accuracy(predictions[task], gold[task])
        elif task == 'MultiRC':
            task_scores[task] = f1a_em(predictions[task], gold[task])
        else:
            task_scores[task] = accuracy(predictions[task], gold[task])
    return sum(task_scores.values()) / len(task_scores)

Common pitfalls

• The average score (Avg) is computed differently across tables: Table 2 averages over 6 tasks, while Table 5 excludes WSC and averages over 5 tasks.
• Small training set sizes for BoolQ and MultiRC cause models to default to predicting the most frequent class, making performance metrics unreliable for those tasks.
• WSC requires human translation and is excluded from machine vs. human translation comparison experiments.

Evidence (verbatim from paper)

Considering the Avg scores in Table 2, the monolingual SloBERTa is the best performing Slovene model. On average, all Slovene BERT models perform better than the Most Frequent baseline.

Citation

@misc{zagar2022slovenesuperglue,
  title={Slovene SuperGLUE Benchmark: Translation and Evaluation},
  author={Žagar et al. (2022)},
  year={2022},
  note={arXiv:2202.04994}
}

• arXiv: 2202.04994