# Content Extraction

> Content reading and extraction guidelines — reading URLs, scraping pages, crawling sites, extracting PDFs, and taking screenshots. Use when reading web content, extracting structured data from pages, or processing documents.

- Skill: `agents-store/content-extraction` (Agent Skill)
- Install (CLI): `npx skillmds@latest add agents-store/content-extraction`
- Raw SKILL.md: https://api.skillmd.com/api/skills/agents-store/content-extraction/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: agents-store (https://skillmd.com/u/agents-store)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/agents-store/content-extraction

---


# Content Extraction

Reading and extracting content from web pages, PDFs, and entire sites. All calls use `~~capability` with fallback (see CONNECTORS.md).

## Reading a Single URL — `~~scrape`

### Fallback chain:
```
1. Jina — fast, clean markdown
2. Firecrawl — JS rendering, advanced options
On error → next provider.
```

### Basic usage:
```
~~scrape("https://example.com/article")
→ Returns clean markdown content
```

### For JS-heavy pages:
```
If ~~scrape returns empty, try provider with waitFor/JS rendering support.
```

## Reading Multiple URLs — `~~batch_scrape`

### Fallback chain:
```
1. Jina parallel — batch all URLs at once
2. Sequential ~~scrape per URL individually
```

### Workflow: search → rank → read
```
1. ~~search(query) → list of URLs
2. Rank by relevance → top-5
3. ~~batch_scrape(top_5_urls) → content
```

## Crawling a Site — `~~crawl`

### Fallback chain:
```
1. Firecrawl crawl → crawl_id, then poll status until "completed"
2. Firecrawl map → list of URLs, then ~~batch_scrape(selected_urls)
```

### With filtering:
```
Crawl with limit, includePaths, excludePaths to control scope.
```

## Structured Data Extraction — `~~extract`

### Fallback chain:
```
1. Firecrawl extract — LLM extraction with JSON schema
2. Firecrawl scrape with JSON options
```

### With JSON schema:
```
~~extract(urls, prompt, schema) → structured data matching your schema
```

## Browser Sessions

For dynamic pages requiring interaction (Firecrawl only):
```
1. Create browser session → session_id
2. Execute browser actions (navigate, click, type, screenshot)
3. Delete browser session
```

## PDF Processing

```
Extract figures, tables, equations from PDF documents.
Useful for academic papers from arXiv or any PDF URL.
```

## Autonomous Research Agent

For complex multi-step research (Firecrawl only):
```
1. Start agent with prompt and optional URLs → agent_id
2. Poll agent status until "completed" (may take 1-5 minutes)
```

## Utility Tools

| Tool | Purpose |
|------|---------|
| Screenshot | Capture a webpage for visual reports |
| Date detection | Detect page publication date for freshness |

## Best Practices

1. **Always start with ~~scrape** — uses fallback chain automatically
2. **JS-heavy pages** — use provider with waitFor/JS rendering
3. **3+ URLs** — use ~~batch_scrape for parallel reading
4. **Large sites** — ~~crawl with limit and depth
5. **PDFs** — use PDF extraction for academic papers
6. **Structured data** — ~~extract with JSON schema
7. **Check dates** — detect page date to filter stale content
8. **Browser sessions** — only for SPA, always delete session after use

## Common Errors

| Error | Fix |
|-------|-----|
| Empty content | Try next provider in fallback chain |
| Timeout | Try lighter alternative (map instead of crawl) |
| 403 Forbidden | Try provider with proxy/stealth support |
| PDF extraction failed | Check URL, try ~~scrape instead |
| Crawl stuck | Set limit and depth, or use map + batch_scrape |