Scientific Data Extraction

Scientific Data Extraction Skill

Overview

This skill provides comprehensive guidance for extracting structured data from scientific literature across multiple input formats (PDF, HTML, images, plain text). It auto-detects the scientific domain to recommend specialized tools when appropriate (particularly for chemistry and materials science) and employs a hierarchical extraction approach with multi-method validation for high-confidence results.

When to Use This Skill

Use this skill when you need to:

• Extract numerical data from scientific papers, reports, or documents
• Digitize graphs and plots to recover underlying data points
• Parse tables from PDFs or images into structured formats (CSV, DataFrame, JSON)
• Extract chemical/materials data including properties, reactions, compounds, and structures
• Convert unstructured text to structured JSON or tabular formats
• Validate extracted data through multi-method cross-checking
• Process document batches with consistent extraction methodology

Input Format Detection

The first step is identifying the input format and routing to appropriate tools:

Plain Text (.txt, .md)

• Domain detection via keyword analysis
• NLP-based entity extraction (spaCy, Stanza)
• Regex patterns for structured data (numbers with units, chemical formulas)
• LLM-based structured extraction

HTML (.html, web pages)

• HTML parsing with BeautifulSoup + lxml
• Table detection and extraction
• Text content extraction with structure preservation
• Domain-specific processing after text extraction

PDF (.pdf)

Images (.png, .jpg, .tiff)

Domain Detection

The skill automatically detects scientific domain to apply specialized tools:

Chemistry/Materials Indicators

• Chemical formulas (H2O, NaCl, TiO2)
• SMILES strings, InChI identifiers
• Reaction arrows (→, ⟶, ⇌)
• Property keywords: melting point, bandgap, conductivity, yield, purity
• Material names and IUPAC nomenclature
• Spectroscopic data patterns (NMR shifts, IR peaks)

When Chemistry/Materials Detected

Apply specialized tools:

• ChemDataExtractor v2: Property extraction, entity recognition, table parsing
• OpenChemIE: Reaction extraction from text, tables, and figures
• Domain-specific NER: Chemical named entity recognition

General Scientific Domain

Use general-purpose extraction:

• Standard NLP pipelines
• LLM-based structured extraction
• Template-based parsing

Extraction Method Hierarchy

Apply methods in order of increasing complexity based on requirements:

Level 1: Quick Extraction (Speed Priority)

When to use: Initial exploration, large document batches, simple structured data

# Quick PDF to text with PyMuPDF4LLM
import pymupdf4llm
text = pymupdf4llm.to_markdown("paper.pdf")

# Quick HTML parsing
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_content, 'lxml')
tables = soup.find_all('table')

Expected confidence: Lower, suitable for screening

Level 2: Standard Extraction (Balanced)

When to use: Research-grade extraction, structure preservation needed

# GROBID for structured PDF parsing
import scipdf_parser
article = scipdf_parser.parse_pdf_to_dict("paper.pdf")

# Docling for layout-aware extraction
from docling.document_converter import DocumentConverter
converter = DocumentConverter()
result = converter.convert("paper.pdf")

# Camelot for bordered tables
import camelot
tables = camelot.read_pdf("paper.pdf", flavor='lattice')
df = tables[0].df

Expected confidence: Medium-high

Level 3: Deep Extraction (Accuracy Priority)

When to use: Publication-quality data, domain-specific extraction

# ChemDataExtractor for chemistry documents
from chemdataextractor import Document
doc = Document.from_file("paper.pdf")
records = doc.records

# OpenChemIE for reaction extraction
from openchemie import OpenChemIE
model = OpenChemIE()
reactions = model.extract_reactions_from_text(text)

# Marker-PDF with OCR for scanned documents
from marker.converters.pdf import PdfConverter
converter = PdfConverter()
result = converter("scanned_paper.pdf")

Expected confidence: High

Level 4: LLM-Enhanced Extraction

When to use: Complex figures, ambiguous data, validation needed

# LLM-based structured extraction
prompt = """
Extract all numerical data from this text as JSON:
- Property name
- Value (number only)
- Unit
- Context (what material/compound)

Text: {text}
"""

# LLM vision for graph interpretation
prompt = """
Analyze this graph image and extract:
1. X-axis label and range
2. Y-axis label and range
3. All data points as (x, y) pairs
4. Any error bars or uncertainty indicators
"""

Expected confidence: Highest when combined with validation

Multi-Method Validation Pipeline

For high-confidence results, use multiple extraction methods and validate:

Step 1: Primary Extraction

Select method based on input type and domain, extract structured data.

Step 2: Secondary Extraction

Run alternative method on same source, compare results and flag discrepancies.

Step 3: LLM Verification Queries

Ask targeted questions to verify extracted data:

• "Is this value X consistent with the context Y?"
• "Does unit Z make sense for property P?"
• "Are there any missing data points in the expected range?"

Step 4: Confidence Scoring

confidence = {
    "score": 0.0,  # 0-1 scale
    "level": "HIGH|MEDIUM|LOW|REVIEW",
    "methods_agreed": [],  # List of methods that produced same result
    "discrepancies": [],   # Any disagreements between methods
    "verification_notes": ""  # LLM verification outcome
}

# Scoring rules:
# - Single method: max 0.7
# - Two methods agree: 0.8
# - Two methods + LLM verification: 0.9
# - Multiple methods + LLM + database cross-reference: 0.95+

Step 5: Database Cross-Reference (Optional)

For chemistry/materials, compare against known databases:

• Materials Project
• AFLOW
• PubChem
• NIST databases

Flag significant deviations from expected ranges.

Output Format

Structure extracted data consistently:

{
  "extraction_metadata": {
    "source": "path/to/document.pdf",
    "source_type": "pdf",
    "domain_detected": "chemistry",
    "methods_used": ["grobid", "chemdataextractor", "llm_verification"],
    "timestamp": "2025-01-18T..."
  },
  "extracted_data": [
    {
      "data_type": "material_property",
      "entity": "TiO2",
      "property": "bandgap",
      "value": 3.2,
      "unit": "eV",
      "source_location": {
        "page": 4,
        "section": "Results",
        "table_id": "Table 2",
        "row": 3
      },
      "confidence": {
        "score": 0.95,
        "level": "HIGH",
        "methods_agreed": ["chemdataextractor", "llm_extraction"],
        "verification_notes": "Value consistent with literature range 3.0-3.4 eV"
      }
    }
  ],
  "validation_summary": {
    "total_extracted": 47,
    "high_confidence": 38,
    "medium_confidence": 7,
    "needs_review": 2,
    "discrepancies": []
  }
}

Step-by-Step Instructions

For PDF Data Extraction

1. Identify document type: Scanned or text-based PDF
2. Choose extraction level: Based on accuracy requirements
3. Detect domain: Check for chemistry/materials indicators
4. Extract text/structure: Use appropriate tool from hierarchy
5. Extract tables separately: Use Camelot, Tabula, or pdfplumber
6. Apply domain tools: If chemistry detected, use ChemDataExtractor
7. Validate: Run secondary extraction or LLM verification
8. Format output: Structure as JSON with confidence scores

For Graph/Plot Digitization

1. Assess graph quality: Resolution, clarity, labeling
2. Identify graph type: Line plot, scatter, bar chart, contour
3. Choose method:
   • Simple, clear graphs: WebPlotDigitizer (manual calibration)
   • Complex or batch: LLM vision extraction
4. Calibrate axes: Define coordinate system
5. Extract data points: Manual selection or automatic detection
6. Validate: Check extracted points against visual inspection
7. Export: CSV or JSON format with uncertainty estimates

For Table Extraction

1. Identify table type: Bordered (lattice) or borderless (stream)
2. Choose tool:
   • Bordered: Camelot with flavor='lattice'
   • Borderless: Tabula or Camelot with flavor='stream'
   • Complex: pdfplumber for fine-grained control
3. Extract to DataFrame: Review structure and headers
4. Clean data: Fix merged cells, missing values, formatting
5. Apply domain parsing: Convert units, parse chemical formulas
6. Validate: Compare against source visually
7. Export: CSV, JSON, or integrate into dataset

For Chemistry/Materials Extraction

1. Confirm domain: Verify chemistry/materials content
2. Choose specialized tool:
   • Properties: ChemDataExtractor v2
   • Reactions: OpenChemIE
   • Structures from images: OSRA or DECIMER
3. Configure extraction: Set up parsers for target properties
4. Run extraction: Process document with domain tools
5. Post-process: Normalize units, standardize identifiers
6. Cross-reference: Compare against databases (Materials Project, PubChem)
7. Validate: LLM verification of unusual values
8. Export: Structured JSON with confidence scores

Best Practices

1. Always start with format detection - Correct tool selection depends on accurate format identification

2. Use the simplest method that works - Start at Level 1 and escalate only if needed

3. Preserve source location - Track page numbers, sections, table IDs for traceability

4. Validate unusual values - Any value outside expected ranges should be flagged and verified

5. Document extraction methodology - Record which tools and settings produced each data point

6. Handle uncertainty explicitly - Include error bounds when available, note when values are approximate

7. Cross-reference chemistry data - Always compare against known databases for sanity checking

8. Use LLM verification judiciously - Most valuable for complex figures and ambiguous cases

Requirements

Core Python Packages

• pymupdf4llm: Quick PDF extraction
• pdfplumber: Detailed PDF analysis
• camelot-py: Table extraction (requires ghostscript)
• beautifulsoup4, lxml: HTML parsing
• spacy: NLP processing
• pandas: Data manipulation

Domain-Specific (Chemistry)

• chemdataextractor: Chemistry NLP (v2 recommended)
• openchemie: Reaction extraction

Optional

• tabula-py: Table extraction (requires Java)
• grobid (server): Academic PDF parsing
• docling: IBM document converter
• marker-pdf: OCR-capable PDF conversion
• tesseract or surya: OCR engines

Limitations

1. Scanned documents require OCR - Quality depends on scan resolution and OCR accuracy

2. Complex table structures - Merged cells, nested headers may require manual correction

3. Graph digitization is approximate - Precision limited by image resolution and calibration

4. Domain tools are specialized - Chemistry tools won't work well on biology or physics texts

5. LLM extraction can hallucinate - Always validate with source or alternative method

6. Some PDFs are protected - May not be extractable due to DRM or image-only content

Related Skills

• literature-review: For systematic literature searching and synthesis
• scientific-reviewer: For evaluating extracted data quality
• materials-databases: For cross-referencing extracted chemistry/materials data
• python-plotting: For visualizing extracted data

References

See the references/ directory for detailed documentation on:

• pdf-tools.md: Comprehensive PDF extraction tool comparison
• table-extraction.md: Table extraction methods and code examples
• graph-digitization.md: Graph data extraction techniques
• chemistry-tools.md: ChemDataExtractor and OpenChemIE usage
• llm-extraction.md: LLM-based extraction patterns and validation

See the examples/ directory for complete workflows:

• extract-from-pdf.md: End-to-end PDF extraction example
• extract-table-data.md: Table extraction comparison
• digitize-graph.md: Graph digitization guide
• chemistry-extraction.md: Chemistry-specific extraction workflow