Portulan Extraglue Eval

portulan-extraglue-eval

PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguese — Tomás Freitas Osório et al. (2024) (arXiv:2404.05333, 2024)

What this evaluates

Evaluates neural models on a Portuguese-language benchmark derived from English GLUE and SuperGLUE tasks, probing capabilities in grammatical acceptability, sentiment, paraphrase detection, semantic similarity, natural language inference, reading comprehension, and causal reasoning.

Datasets

• CoLA — total ?; splits: train (-1), val (-1), test (-1)
• SST-2 — total ?; splits: train (-1), val (-1), test (-1)
• MRPC — total ?; splits: train (-1), val (-1), test (-1)
• QQP — total ?; splits: train (-1), val (-1), test (-1)
• STS-B — total ?; splits: train (-1), val (-1), test (-1)
• WiC — total ?; splits: train (-1), val (-1), test (-1)
• MNLI — total ?; splits: train (-1), val (-1), test (-1)
• QNLI — total ?; splits: train (-1), val (-1), test (-1)
• RTE — total ?; splits: train (-1), val (-1), test (-1)
• WNLI — total ?; splits: train (-1), val (-1), test (-1)
• WSC — total ?; splits: train (-1), val (-1), test (-1)
• CB — total ?; splits: train (-1), val (-1), test (-1)
• AXb — total ?; splits: test (-1)
• AXg — total ?; splits: test (-1)
• BoolQ — total ?; splits: train (-1), val (-1), test (-1)
• MultiRC — total ?; splits: train (-1), val (-1), test (-1)
• ReCoRD — total ?; splits: train (-1), val (-1), test (-1)
• COPA — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• single-number performance metric (primary) — range: percent
  • Aggregated score across all tasks, typically computed as accuracy for classification tasks, F1 for paraphrase/QQP, and Pearson correlation for STS-B, following standard GLUE/SuperGLUE protocols.
• similarity score — range: [1, 5]
  • Continuous score from 1 to 5 predicting semantic similarity for sentence pairs in STS-B.

Input / output format

Input: Sentence pairs, single sentences, or context-question-answer triples depending on the specific GLUE/SuperGLUE task variant.

Output: Class labels (e.g., acceptability, sentiment, entailment/contradiction/neutral, yes/no), similarity scores (1-5), or selected entities/answers.

Scoring recipe

def compute_glue_score(predictions, gold_labels, task_type):
    if task_type in ['classification', 'NLI', 'QA']:
        return sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    elif task_type == 'paraphrase':
        return f1_score(gold_labels, predictions, average='binary')
    elif task_type == 'STS':
        return pearsonr(gold_labels, predictions)[0]
    return 0.0

# Aggregate across all PORTULAN ExtraGLUE tasks
final_score = mean([compute_glue_score(preds, golds, t) for t, preds, golds in tasks])

Common pitfalls

• Translation-induced linguistic artifacts, such as gender bias and idiom misrepresentation, may skew performance on Portuguese variants compared to English baselines.
• The benchmark aggregates diverse task types (classification, regression, NLI, QA) into a single score, which can mask poor performance on specific linguistic phenomena.
• Low-rank adaptation (LoRA) fine-tuning on Albertina may underfit complex reasoning tasks compared to full fine-tuning, affecting cross-task comparability.

Evidence (verbatim from paper)

Both GLUE and SuperGLUE are aggregations of existing public datasets accompanied by a single-number performance metric and an analysis toolkit. The Semantic Textual Similarity Benchmark (STS-B)G (Cer et al.,, [2017])* is a task for predicting a similarity score (from 1 to 5) for each sentence pair.

Citation

@misc{osorio2024portulanextraglue,
  title={PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguese},
  author={Tomás Freitas Osório et al. (2024)},
  year={2024},
  note={arXiv:2404.05333}
}

• arXiv: 2404.05333