qhjqhj00 scientific-topic-classification-eval
SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics — You et al. (2024) (arXiv:2410.01946, 2024)
What this evaluates
Evaluates the ability of language models to accurately classify scientific abstracts into fine-grained disciplinary or sub-disciplinary categories under few-shot and zero-shot conditions.
Datasets
- SDPRA 2021 — total ?; splits: test (-1)
- arXiv — total ?; splits: test (-1)
- S2ORC — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly predicted class labels out of the total number of test instances. Reported as the mean accuracy across five random seeds/iterations.
Input / output format
Input: Scientific abstract text (English).
Output: Predicted class label (topic/category name).
Scoring recipe
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = (correct / len(gold_labels)) * 100
return accuracy
Common pitfalls
- Few-shot experiments are conducted with varying shot counts (1, 5, 10, 20, 50) and must be averaged over five random seeds to match reported results.
- RetroPrompt is excluded from the 1-shot setting because it requires at least two labeled examples for tuning.
- Zero-shot evaluation uses approximately 10% of each dataset held out for testing, not the full test split.
Evidence (verbatim from paper)
We evaluate model performance across five random seeds to account for variability Hu et al. ([2021]); Ding et al. ([2022b]). ... conducting tests with 1, 5, 10, 20, and 50 shots across all datasets and reporting accuracy as an evaluation metric.
Citation
@misc{you2024sciprompt,
title={SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics},
author={You et al. (2024)},
year={2024},
note={arXiv:2410.01946}
}
- arXiv: 2410.01946