qhjqhj00 rigour-classifier-eval
On the Rigour of Scientific Writing: Criteria, Analysis, and Insights — James et al. (2024) (arXiv:2410.04981, 2024)
What this evaluates
Evaluates a classifier's ability to predict the scientific rigour of academic papers (rated 4* vs non-4*) based solely on their abstracts and introductions. The setup tests whether linguistic patterns in early paper sections correlate with institutional rigour ratings.
Datasets
- REF dataset — total ?; splits: train (-1), test (-1)
- ICLR dataset — total ?; splits: test (-1)
- ACL dataset — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: [0, 1]- Standard binary classification accuracy: the proportion of correctly predicted rigour labels (4* vs non-4*) out of the total instances.
Input / output format
Input: Raw text of the abstract and introduction sections from scientific papers.
Output: Binary classification label: 4* (high rigour) or non-4* (low rigour).
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return correct / len(gold_labels)
Common pitfalls
- Ground truth relies on coarse institutional REF 2021 ratings rather than fine-grained paper-level annotations.
- Evaluation is restricted to abstracts and introductions, which may not capture the full methodological rigour presented in later sections.
- Automatic extraction via pattern matching on section titles may miss or misalign content in papers with non-standard formatting.
Evidence (verbatim from paper)
The submissions from UOA 11 ... was used to create the dataset. ... we collected publications from institutions whose outputs are predominantly rated as 4* (e.g., Imperial College London and Oxford), as well as from institutions whose outputs are predominantly rated as non-4*, to form a binary labelled dataset for rigour. ... to show the effectiveness of our rigour classifier on papers submitted after 2021.
Citation
@misc{james2024rigour,
title={On the Rigour of Scientific Writing: Criteria, Analysis, and Insights},
author={James et al. (2024)},
year={2024},
note={arXiv:2410.04981}
}
- arXiv: 2410.04981