Confready Checklist Eval

confready-checklist-eval

ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses — Galarnyk et al. (2024) (arXiv:2408.04675, 2024)

What this evaluates

This benchmark evaluates a model's ability to accurately answer conference submission checklist questions based on manuscript content. It specifically probes long-form document understanding, retrieval-augmented generation (RAG) effectiveness, and the model's capacity to reflect on ethical considerations, reproducibility, and societal impacts.

Datasets

• ConfReady Evaluation Set — total 93; splits: test (93); repo https://github.com/gtfintechlab/ConfReady

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered checklist questions compared to human-annotated references. Calculated as (number of correct responses / total number of responses) × 100.

Input / output format

Input: Manuscript provided in TeX or Markdown format, paired with a specific conference checklist question.

Output: A generated response (typically a yes/no answer with justification) for each checklist question.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, references):
    if normalize(pred) == normalize(gold):
        correct += 1
accuracy = (correct / len(references)) * 100

Common pitfalls

• Models often require explicit mentions of all required details (e.g., parameters, GPU hours, infrastructure) to answer 'YES', whereas human annotators accept section references (e.g., 'Section 4') as sufficient.
• Longer Main conference submissions are significantly harder to answer accurately than Findings papers due to increased length and complexity.
• Converting PDFs to Markdown for LM input often causes structural extraction errors that disrupt retrieval and alignment, making TeX parsing preferable.

Evidence (verbatim from paper)

Table 2: Accuracy comparison of RAG, LMs on TeX, and LMs on PDFs for ACL Main (Long) and ACL Findings (Long) papers. We evaluate Llama-3.1–405B, Llama-3.3–70B Meta et al. ([2024]), and GPT-4o OpenAI ([2023a]) on the evaluation sample. Due to compute limits, experiments focus on long-form ACL submissions. Human-annotated answers serve as references, allowing us to evaluate how effectively models can reflect on ethical considerations, reproducibility, and societal impacts in each setup.

Citation

@misc{galarnyk2024confready,
  title={ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses},
  author={Galarnyk et al. (2024)},
  year={2024},
  note={arXiv:2408.04675}
}

• arXiv: 2408.04675