Pdfqa Eval

pdfqa-eval

pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs — Schimanski et al. (2026) (arXiv:2601.02285, 2026)

What this evaluates

Evaluates end-to-end question answering over PDF documents, probing parsing, retrieval, and reasoning capabilities across diverse document types, modalities, and complexity dimensions.

Datasets

• pdfQA — total 4000; splits: test (4000); repo https://github.com/tobischimanski/pdfQA

Metrics

• G-Eval correctness (primary) — range: [1, 5]
  • LLM-as-judge evaluation using a G-Eval prompt that rates answer correctness on a 1–5 scale, where 5 indicates a completely correct answer.

Input / output format

Input: Parsed text extracted from the source PDF using PyMUPDF, concatenated with the corresponding question.

Output: Free-form text answer generated by the model, which is then scored by an LLM judge.

Scoring recipe

def compute_metric(predictions, questions, contexts):
    scores = []
    for pred, q, ctx in zip(predictions, questions, contexts):
        prompt = f'Question: {q}\nContext: {ctx}\nAnswer: {pred}\nRate correctness 1-5:'
        score = llm_as_judge(prompt)  # Returns int 1-5
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• The evaluation relies entirely on an LLM-as-judge (G-Eval) rather than exact match or human verification, which can introduce scoring bias or inconsistency.
• PDF parsing is done naively by extracting all text via PyMUPDF without preserving layout or structure, potentially disadvantaging models that rely on visual or structural cues.
• Averaging scores across heterogeneous complexity dimensions (e.g., modality, file length, reasoning depth) can mask severe performance drops on specific subtasks like table interpretation or long-document retrieval.

Evidence (verbatim from paper)

Then, we use the G-Eval correctness prompt in Figure [8] to evaluate the answer. This prompt scales the answer between 1-5, where 5 indicates a completely correct answer (see App. [J]).

Citation

@misc{schimanski2026pdfqa,
  title={pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs},
  author={Schimanski et al. (2026)},
  year={2026},
  note={arXiv:2601.02285}
}

• arXiv: 2601.02285