Superglue Eval

superglue-eval

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems — Wang et al. (2019) (arXiv:1905.00537, 2019)

What this evaluates

Evaluates general-purpose language understanding across eight diverse tasks including coreference resolution, question answering, and natural language inference. It probes a model's ability to handle complex reasoning, sample-efficient learning, and transfer learning beyond standard GLUE capabilities.

Datasets

• SuperGLUE — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• SuperGLUE score (primary) — range: percent
  • The average of task-specific scores across all benchmark tasks. Most tasks use accuracy, while ReCoRD uses F1. Scores are reported as points out of 100.

Input / output format

Input: Natural language text formatted as task-specific sequences. For sentence-pair tasks, sentences are concatenated with a [SEP] token. For QA tasks, context and answer choices are concatenated. For span-based tasks, word representations are extracted from the input sequence.

Output: Task-specific predictions: classification labels for BoolQ, CB, RTE, WiC; scalar scores for COPA choices; logistic regression probabilities for MultiRC answer choices; candidate probabilities for ReCoRD; and span labels for WSC.

Scoring recipe

def compute_superglue_score(predictions, golds, task):
    if task == 'ReCoRD':
        return f1_score(predictions, golds)
    elif task == 'MultiRC':
        return multi_rc_accuracy(predictions, golds)
    else:
        return accuracy(predictions, golds)

task_scores = [compute_superglue_score(preds[g], golds[g], g) for g in tasks]
avg_score = sum(task_scores) / len(task_scores)
return avg_score

Common pitfalls

• WSC performance can drop below simple baselines due to the extremely small dataset size and lack of data augmentation.
• Human performance estimates vary across tasks and are sometimes approximated via crowdworkers, introducing potential measurement noise.
• Winogender diagnostics show near-perfect scores for all models, but this is an artifact of random guessing rather than true gender parity.

Evidence (verbatim from paper)

Using BERT increases the average SuperGLUE score by 25 points, attaining significant gains on all of the benchmark tasks, particularly MultiRC, ReCoRD, and RTE. On WSC, BERT actually performs worse than the simple baselines, likely due to the small size of the dataset and the lack of data augmentation.

Citation

@misc{wang2019superglue,
  title={SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems},
  author={Wang et al. (2019)},
  year={2019},
  note={arXiv:1905.00537}
}

• arXiv: 1905.00537