Datarubrics Eval

datarubrics-eval

Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability — Winata et al. (2025) (arXiv:2506.01789, 2025)

What this evaluates

Evaluates the quality, accountability, and documentation standards of dataset and benchmark papers across major AI conferences. It probes whether papers provide transparent data collection guidelines, quality assurance practices, and clear provenance using a structured rubric.

Datasets

• Conference Dataset & Benchmark Papers (2021-2024) — total ?; splits: evaluation_corpus (-1); repo https://github.com/datarubrics/datarubrics

Metrics

• datarubrics_compliance_rate (primary) — range: percent
  • Proportion of papers that satisfy each dimension of the DataRubrics framework. Calculated as (number of papers meeting criterion) / (total papers evaluated). Also reports human-LLM annotation agreement rate.

Input / output format

Input: PDF documents of academic papers, including titles, abstracts, and full text extracted via OCR, along with conference/year metadata.

Output: Structured JSON containing predicted labels, reasoning, and references for each rubric category, plus aggregate compliance percentages per conference/year.

Scoring recipe

# For each paper in the corpus:
# 1. Extract text via OCR (OlmOCR)
# 2. Prompt LLM (GPT-4.1-mini) with rubric template to generate labels/reasoning
# 3. Human experts annotate a subset (100 NeurIPS papers) with QA review
# 4. Compute compliance_rate = sum(paper.meets_rubric_criterion) / total_papers
# 5. Compute agreement = sum(human_label == llm_label) / total_annotated

Common pitfalls

• Human annotators frequently overlook fine-grained details, leading to ~26% residual error even after QA.
• Misclassification between human-written and model-generated annotations skews compliance rates.
• OCR extraction can introduce noise from anonymization artifacts or embedded tables, affecting LLM judgment.

Evidence (verbatim from paper)

To evaluate the usefulness of the metric and the assessment pipeline, we engaged domain experts to perform rubric-based annotations. ... We evaluate responses using a proprietary model GPT-4.1-mini. ... As shown in Figure 7, 26% of the annotations remain incorrect, despite having passed QA by human annotators tasked with identifying whether a paper includes specific annotations.

Citation

@misc{winata2025datarubrics,
  title={Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability},
  author={Winata et al. (2025)},
  year={2025},
  note={arXiv:2506.01789}
}

• arXiv: 2506.01789