PDF Parsing Chunking Eval

pdf-parsing-chunking-eval

Empirical Evaluation of PDF Parsing and Chunking for Financial Question Answering with RAG — El Bachyr et al. (2026) (arXiv:2604.12047, 2026)

What this evaluates

This evaluation probes the retrieval accuracy of RAG pipelines when processing financial PDFs, specifically testing how different PDF parsers, chunking strategies, and overlap percentages affect the retrieval of relevant pages for both narrative text and structured table queries.

Datasets

• FinanceBench — total 168; splits: test (168)
• TableQuest — total 169; splits: test (169)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of the reciprocal of the rank of the first relevant document for each query. Calculated as 1/rank.
• P@1 — range: [0, 1]
  • Precision at rank 1: 1 if the top-ranked retrieved page is relevant to the query, 0 otherwise.
• R@3 — range: [0, 1]
  • Recall at rank 3: the fraction of all relevant pages that appear within the top 3 retrieved results.

Input / output format

Input: Natural language query (text or table-focused) and a set of PDF documents parsed into text chunks/pages.

Output: A ranked list of retrieved pages/chunks, evaluated at top-1 and top-3 positions.

Scoring recipe

def compute_metrics(retrieved_pages, relevant_pages):
    p_at_1 = 1.0 if retrieved_pages[0] in relevant_pages else 0.0
    r_at_3 = len(set(retrieved_pages[:3]) & relevant_pages) / len(relevant_pages)
    reciprocal_rank = 0.0
    for i, page in enumerate(retrieved_pages):
        if page in relevant_pages:
            reciprocal_rank = 1.0 / (i + 1)
            break
    return p_at_1, r_at_3, reciprocal_rank

Common pitfalls

• Assuming larger overlap (50%) always improves retrieval; the study shows 25% is optimal for balancing context and index size.
• Treating all chunking strategies equally; structure-aware chunkers (semantic, SDPM) increase index overhead without consistently improving top-rank accuracy compared to neural or sentence chunking.
• Ignoring parser-table interaction; naive parsers like PyPDF2 fail on structured tables, making parser choice critical for table-focused QA.

Evidence (verbatim from paper)

Retrieval performance is measured by $P@1$, $R@3$, and MRR. Text is extracted using six PDF parsers (pdfminer, PyMuPDF, PyPDF2, Unstructured, pdfplumber, pypdfium2) and split with six chunking strategies (token, sentence, semantic, recursive, SDPM, neural). To ensure fair comparisons, we fix the chunk size at 512 tokens with $25%$ (128 tokens) overlap to provide sufficient context while minimizing fragmentation effects resulting from text chunking.

Citation

@misc{elbachyr2026empiricalevaluation,
  title={Empirical Evaluation of PDF Parsing and Chunking for Financial Question Answering with RAG},
  author={El Bachyr et al. (2026)},
  year={2026},
  note={arXiv:2604.12047}
}

• arXiv: 2604.12047