blimp-glue-superglue-eval
PaPaformer: Language Model from Pre-trained Parallel Paths — Tapaninaho et al. (2025) (arXiv:2508.00544, 2025)
What this evaluates
Evaluates language understanding, linguistic acceptability, sentiment analysis, natural language inference, and factual reasoning under low-resource fine-tuning conditions.
Datasets
Metrics
accuracy (primary) — range: [0, 1]
- Standard task-level accuracy calculated as the proportion of correct predictions. Scores are averaged across multiple random seeds (typically 5) to mitigate initialization variance.
Input / output format
Input: Text sequences or sentence pairs from BLiMP, GLUE, and SuperGLUE tasks, formatted according to the babylm/evaluation-pipeline-2024.
Output: Predicted class labels for each task.
Scoring recipe
import numpy as np
def compute_accuracy(predictions, gold_labels):
return np.mean(np.array(predictions) == np.array(gold_labels))
seed_scores = []
for seed in seeds:
preds, golds = run_finetune_and_evaluate(seed)
seed_scores.append(compute_accuracy(preds, golds))
final_score = np.mean(seed_scores)
Common pitfalls
- High sensitivity to random weight initialization seeds, especially for small models and small datasets, causing up to 12% metric variance.
- Routing/path selection in parallel architectures does not consistently align with optimal task performance; dominant path usage often correlates better than balanced routing.
- Qualitative generation evaluation was restricted to narrative prompts due to poor model performance on math/instructional prompts.
Evidence (verbatim from paper)
Model evaluations closely followed the methodologies provided in the babylm/evaluation-pipeline-2024 (Choshen et al., [2024]). However, not all tasks were included in the more lightweight evaluation pipeline, which this work followed. This pipeline applied the full BLiMP (Warstadt et al., [2020]) benchmark and a selected subset of tasks from the GLUE (Wang et al., [2018]) and SuperGLUE (Wang et al., [2019]) benchmarks... To address this problem, the final evaluation included multiple fine-tuning runs (5 different seeds: 3, 12, 42, 100, 200) for each model, and the reported results are averaged scores across runs. ... Table[6] exhibits an interesting perspective of the selection and utilization of parallel-model path... Accuracy | --- | 40% | 46.5% | 50
Citation
@misc{tapaninaho2025papapformer,
title={PaPaformer: Language Model from Pre-trained Parallel Paths},
author={Tapaninaho et al. (2025)},
year={2025},
note={arXiv:2508.00544}
}
1---2name: blimp-glue-superglue-eval3description: Evaluates language understanding, linguistic acceptability, sentiment analysis, natural language inference, and factual reasoning under low-resource fine-tuning conditions. Use when the user wants to benchmark on BLiMP, GLUE (subset), SuperGLUE (subset), or asks about evaluating this task. Reports accuracy.4---56# blimp-glue-superglue-eval78> PaPaformer: Language Model from Pre-trained Parallel Paths — Tapaninaho et al. (2025) (arXiv:2508.00544, 2025)910## What this evaluates1112Evaluates language understanding, linguistic acceptability, sentiment analysis, natural language inference, and factual reasoning under low-resource fine-tuning conditions.1314## Datasets1516- **BLiMP, GLUE (subset), SuperGLUE (subset)** — total ?; splits: train (-1), val (-1), test (-1); HF `blimp, glue, super_glue`; repo https://github.com/babylm/evaluation-pipeline-20241718## Metrics1920- `accuracy` **(primary)** — range: [0, 1]21 - Standard task-level accuracy calculated as the proportion of correct predictions. Scores are averaged across multiple random seeds (typically 5) to mitigate initialization variance.2223## Input / output format2425**Input**: Text sequences or sentence pairs from BLiMP, GLUE, and SuperGLUE tasks, formatted according to the babylm/evaluation-pipeline-2024.2627**Output**: Predicted class labels for each task.2829## Scoring recipe3031```python32import numpy as np33def compute_accuracy(predictions, gold_labels):34 return np.mean(np.array(predictions) == np.array(gold_labels))35seed_scores = []36for seed in seeds:37 preds, golds = run_finetune_and_evaluate(seed)38 seed_scores.append(compute_accuracy(preds, golds))39final_score = np.mean(seed_scores)40```4142## Common pitfalls4344- High sensitivity to random weight initialization seeds, especially for small models and small datasets, causing up to 12% metric variance.45- Routing/path selection in parallel architectures does not consistently align with optimal task performance; dominant path usage often correlates better than balanced routing.46- Qualitative generation evaluation was restricted to narrative prompts due to poor model performance on math/instructional prompts.4748## Evidence (verbatim from paper)4950> Model evaluations closely followed the methodologies provided in the babylm/evaluation-pipeline-2024 (Choshen et al., [2024]). However, not all tasks were included in the more lightweight evaluation pipeline, which this work followed. This pipeline applied the full BLiMP (Warstadt et al., [2020]) benchmark and a selected subset of tasks from the GLUE (Wang et al., [2018]) and SuperGLUE (Wang et al., [2019]) benchmarks... To address this problem, the final evaluation included multiple fine-tuning runs (5 different seeds: 3, 12, 42, 100, 200) for each model, and the reported results are averaged scores across runs. ... Table[6] exhibits an interesting perspective of the selection and utilization of parallel-model path... Accuracy | --- | 40% | 46.5% | 505152## Citation5354```bibtex55@misc{tapaninaho2025papapformer,56 title={PaPaformer: Language Model from Pre-trained Parallel Paths},57 author={Tapaninaho et al. (2025)},58 year={2025},59 note={arXiv:2508.00544}60}61```6263- arXiv: 2508.00544
qhjqhj00