# Slovene Superglue Eval

> Evaluates monolingual, cross-lingual, and multilingual NLP models on a human- and machine-translated Slovene version of the SuperGLUE benchmark. It probes how well models handle morphological and grammatical challenges in low-resource language processing, and compares translation quality impacts on downstream task performance. Use when the user wants to benchmark on Slovene SuperGLUE, or asks about evaluating this task. Reports Avg.

- Skill: `qhjqhj00/slovene-superglue-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/slovene-superglue-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/slovene-superglue-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/slovene-superglue-eval

---


# slovene-superglue-eval

> Slovene SuperGLUE Benchmark: Translation and Evaluation — Žagar et al. (2022) (arXiv:2202.04994, 2022)

## What this evaluates

Evaluates monolingual, cross-lingual, and multilingual NLP models on a human- and machine-translated Slovene version of the SuperGLUE benchmark. It probes how well models handle morphological and grammatical challenges in low-resource language processing, and compares translation quality impacts on downstream task performance.

## Datasets

- **Slovene SuperGLUE** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Avg` **(primary)** — range: percent
  - Average of task-specific scores across the evaluated tasks. Task scores are Accuracy (Acc.), F1/Accuracy (for CB), or F1a/EM (for MultiRC). Note that the set of tasks averaged differs by experiment (6 tasks in Table 2, 5 tasks excluding WSC in Table 5).
- `Accuracy (Acc.)` — range: percent
  - Percentage of correctly predicted class labels for tasks like BoolQ, COPA, RTE, and WSC.
- `F1/Accuracy` — range: percent
  - Reported as F1/Accuracy for the CB task, evaluating both macro-F1 and accuracy.
- `F1a/EM` — range: percent
  - Reported for MultiRC, evaluating average F1 and exact match for multiple-choice answers.

## Input / output format

**Input**: Slovene text instances corresponding to SuperGLUE tasks (BoolQ, CB, COPA, MultiRC, RTE, WSC), including premises, hypotheses, questions, and context as appropriate for each task.

**Output**: Predicted class labels or answer spans for each instance.

## Scoring recipe

```python
def compute_avg(predictions, gold):
    task_scores = {}
    for task in ['BoolQ', 'CB', 'COPA', 'MultiRC', 'RTE', 'WSC']:
        if task == 'CB':
            task_scores[task] = f1_accuracy(predictions[task], gold[task])
        elif task == 'MultiRC':
            task_scores[task] = f1a_em(predictions[task], gold[task])
        else:
            task_scores[task] = accuracy(predictions[task], gold[task])
    return sum(task_scores.values()) / len(task_scores)
```

## Common pitfalls

- The average score (Avg) is computed differently across tables: Table 2 averages over 6 tasks, while Table 5 excludes WSC and averages over 5 tasks.
- Small training set sizes for BoolQ and MultiRC cause models to default to predicting the most frequent class, making performance metrics unreliable for those tasks.
- WSC requires human translation and is excluded from machine vs. human translation comparison experiments.

## Evidence (verbatim from paper)

> Considering the Avg scores in Table 2, the monolingual SloBERTa is the best performing Slovene model. On average, all Slovene BERT models perform better than the Most Frequent baseline.

## Citation

```bibtex
@misc{zagar2022slovenesuperglue,
  title={Slovene SuperGLUE Benchmark: Translation and Evaluation},
  author={Žagar et al. (2022)},
  year={2022},
  note={arXiv:2202.04994}
}
```

- arXiv: 2202.04994