pdf-extraction-eval
A Benchmark of PDF Information Extraction Tools using a Multi-Task and Multi-Domain Evaluation Framework for Academic Documents — Norman Meuschke et al. (arXiv:2303.09957, 2023)
What this evaluates
Evaluates open-source PDF information extraction tools across multiple content elements (metadata, references, tables, paragraphs, sections, etc.) on academic documents. It probes how well different tools handle layout-based segmentation, text extraction, and structural recognition in real-world academic PDFs.
Datasets
Metrics
F1 score (primary) — range: [0, 1]
- Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). For metadata and general extraction tasks, a cumulative F1 score is reported, calculated as the sum of individual F1 scores across sub-elements (e.g., title, abstract, authors).
Input / output format
Input: PDF files of academic documents (primarily from arXiv).
Output: Structured extraction of specific content elements per document, including title, abstract, authors, references, tables, paragraphs, sections, captions, footers, and equations.
Scoring recipe
def compute_f1(preds, gold):
tp = len(set(preds) & set(gold))
fp = len(set(preds) - set(gold))
fn = len(set(gold) - set(preds))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * (prec * rec) / (prec + rec) if (prec + rec) > 0 else 0
def compute_cumulative_f1(element_f1s):
return sum(element_f1s.values())
Common pitfalls
- Cumulative F1 scores can mask poor performance on specific sub-elements (e.g., a tool might score well on title/abstract but poorly on authors).
- Table extraction tools often misidentify two-column layouts as tables or include captions in table regions, artificially lowering F1 scores.
- Tools may fail silently or throw PDF read exceptions, leading to discrepancies between '# Detected' and '# Processed' items in results tables.
Evidence (verbatim from paper)
Figure 6 shows the cumulative $F_{1}$ scores of CERMINE, GROBID, PdFig, and Science Parse for the metadata extraction task, i.e., extracting title, abstract, and authors. Consequently, the best possible cumulative $F_{1}$ score equals three.
Citation
@misc{meuschke2023pdfbenchmark,
title={A Benchmark of PDF Information Extraction Tools using a Multi-Task and Multi-Domain Evaluation Framework for Academic Documents},
author={Norman Meuschke et al.},
year={2023},
note={arXiv:2303.09957}
}
1---2name: pdf-extraction-eval3description: Evaluates open-source PDF information extraction tools across multiple content elements (metadata, references, tables, paragraphs, sections, etc.) on academic documents. It probes how well different tools handle layout-based segmentation, text extraction, and structural recognition in real-world academic PDFs. Use when the user wants to benchmark on DocBank, or asks about evaluating this task. Reports F1 score.4---56# pdf-extraction-eval78> A Benchmark of PDF Information Extraction Tools using a Multi-Task and Multi-Domain Evaluation Framework for Academic Documents — Norman Meuschke et al. (arXiv:2303.09957, 2023)910## What this evaluates1112Evaluates open-source PDF information extraction tools across multiple content elements (metadata, references, tables, paragraphs, sections, etc.) on academic documents. It probes how well different tools handle layout-based segmentation, text extraction, and structural recognition in real-world academic PDFs.1314## Datasets1516- **DocBank** — total ?; splits: test (-1); repo https://github.com/doc-analysis/DocBank1718## Metrics1920- `F1 score` **(primary)** — range: [0, 1]21 - Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). For metadata and general extraction tasks, a cumulative F1 score is reported, calculated as the sum of individual F1 scores across sub-elements (e.g., title, abstract, authors).2223## Input / output format2425**Input**: PDF files of academic documents (primarily from arXiv).2627**Output**: Structured extraction of specific content elements per document, including title, abstract, authors, references, tables, paragraphs, sections, captions, footers, and equations.2829## Scoring recipe3031```python32def compute_f1(preds, gold):33 tp = len(set(preds) & set(gold))34 fp = len(set(preds) - set(gold))35 fn = len(set(gold) - set(preds))36 prec = tp / (tp + fp) if (tp + fp) > 0 else 037 rec = tp / (tp + fn) if (tp + fn) > 0 else 038 return 2 * (prec * rec) / (prec + rec) if (prec + rec) > 0 else 03940def compute_cumulative_f1(element_f1s):41 return sum(element_f1s.values())42```4344## Common pitfalls4546- Cumulative F1 scores can mask poor performance on specific sub-elements (e.g., a tool might score well on title/abstract but poorly on authors).47- Table extraction tools often misidentify two-column layouts as tables or include captions in table regions, artificially lowering F1 scores.48- Tools may fail silently or throw PDF read exceptions, leading to discrepancies between '# Detected' and '# Processed' items in results tables.4950## Evidence (verbatim from paper)5152> Figure 6 shows the cumulative $F_{1}$ scores of CERMINE, GROBID, PdFig, and Science Parse for the metadata extraction task, i.e., extracting title, abstract, and authors. Consequently, the best possible cumulative $F_{1}$ score equals three.5354## Citation5556```bibtex57@misc{meuschke2023pdfbenchmark,58 title={A Benchmark of PDF Information Extraction Tools using a Multi-Task and Multi-Domain Evaluation Framework for Academic Documents},59 author={Norman Meuschke et al.},60 year={2023},61 note={arXiv:2303.09957}62}63```6465- arXiv: 2303.09957
qhjqhj00