qhjqhj00 onlysports-benchmark-eval
OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameters — Chen et al. (2024) (arXiv:2409.00286, 2024)
What this evaluates
Evaluates a sports-domain language model's generation capability on sports-specific tasks and its zero-shot commonsense reasoning performance on general benchmarks.
Datasets
- OnlySports Benchmark — total ?; splits: test (-1)
- HellaSwag — total ?; splits: test (-1); HF
hellaswag - PIQA — total ?; splits: test (-1); HF
piqa - ARC-challenge — total ?; splits: test (-1); HF
ai2_arc - ARC-easy — total ?; splits: test (-1); HF
ai2_arc
Metrics
OS-acc(primary) — range: percent- Accuracy on sports domain generation tasks, calculated as the proportion of correctly generated or selected outputs.
OS-rel— range: percent- Relation/extraction accuracy or score for sports domain tasks.
OS-Avg.— range: percent- Average score across OS-acc and OS-rel.
ARC-e— range: percent- Accuracy on ARC-easy multiple-choice questions.
ARC-c— range: percent- Accuracy on ARC-challenge multiple-choice questions.
PIQA— range: percent- Accuracy on Physical Interaction Question Answering.
HS— range: percent- Accuracy on HellaSwag completion task.
Input / output format
Input: Zero-shot prompts for sports domain generation tasks and commonsense reasoning questions.
Output: Model-generated text completions or selected answer choices.
Scoring recipe
def calculate_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
return correct / len(gold) * 100
# For multiple-choice benchmarks (ARC, PIQA, HS):
# predictions are model-generated choices or extracted answers.
# gold are ground-truth labels.
# Score is percentage of exact matches.
Common pitfalls
- Baselines' general benchmark scores are retrieved directly from their original papers rather than re-evaluated on the same setup.
- Training was truncated at 315B tokens due to funding constraints, so results do not reflect convergence on the full 600B-token dataset.
- Evaluation is strictly zero-shot without instruction tuning or fine-tuning, which may disadvantage models trained with instruction data.
Evidence (verbatim from paper)
We compare the final OnlySportsLM checkpoint on OnlySports Benchmark and zero-shot commonsense reasoning tasks (Hellaswag, PIQA, ARC-challenge, and ARC-easy) with previous training checkpoints and recent open-source models. Table 3 compares our OnlySportsLM and two recent state-of-the-art general-purpose models, ranging from 137M to 1.7B parameters... OS-acc | OS-rel | OS-Avg. | ARC-e | ARC-c | PIQA | HS
Citation
@misc{chen2024onlysportslm,
title={OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameters},
author={Chen et al. (2024)},
year={2024},
note={arXiv:2409.00286}
}
- arXiv: 2409.00286