Commitment Audit Eval

commitment-audit-eval

Commitment Checklist: Auditing Author Commitments in Peer Review — Chung-Chi Chen et al. (2026) (arXiv:2603.00003, 2026)

What this evaluates

Evaluates the extent to which authors fulfill promises made during peer review rebuttals in their final camera-ready papers, and classifies unfulfilled commitments by severity and difficulty.

Datasets

• ICLR 2025 — total 7192; splits: test (7192)
• EMNLP 2024 — total 1050; splits: test (1050)

Metrics

• fulfillment rate (primary) — range: percent
  • The percentage of extracted author commitments that are successfully implemented or addressed in the final camera-ready manuscript version.

Input / output format

Input: Rebuttal texts containing explicit author commitments and the corresponding final camera-ready paper versions.

Output: A binary fulfillment label (fulfilled/unfulfilled) for each commitment, plus a quadrant classification (severity/difficulty) for unfulfilled commitments.

Scoring recipe

def calculate_fulfillment_rate(predictions, gold):
    fulfilled = sum(1 for p, g in zip(predictions, gold) if p == g == 'fulfilled')
    total = len(gold)
    return (fulfilled / total) * 100

Common pitfalls

• Non-fulfillment does not always indicate bad faith; authors may have attempted experiments that yielded negative or inconclusive results.
• The short post-acceptance revision window often makes initial promises unrealistic or premature.
• LLM-based extraction and classification rely on specific prompts detailed in the appendix, which may affect reproducibility.

Evidence (verbatim from paper)

4.2 Fulfillment Rates

What fraction of the commitments were actually fulfilled in the final versions? Overall, we found that the majority of promises were kept, but a significant minority were not: ICLR 2025: Approximately 75.89% of the extracted commitments were fulfilled in the camera-ready papers, leaving about 24% unfulfilled. EMNLP 2024: About 72.91% of commitments were fulfilled, with around 27% unfulfilled.

Citation

@misc{chen2026commitmentchecklist,
  title={Commitment Checklist: Auditing Author Commitments in Peer Review},
  author={Chung-Chi Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.00003}
}

• arXiv: 2603.00003