Content Extraction
Reading and extracting content from web pages, PDFs, and entire sites. All calls use ~~capability with fallback (see CONNECTORS.md).
Reading a Single URL — ~~scrape
Fallback chain:
1. Jina read_url — fast, clean markdown (pages and PDFs)
2. Firecrawl scrape — JS rendering, advanced options
3. Exa fetch (pass maxCharacters: 20000 — the default is 3000 per page) — last resort
On error → next provider.
Basic usage:
~~scrape("https://example.com/article")
→ Returns clean markdown content
Cheap mode — only the relevant passages:
~~scrape(url, question: "What are the default rate limits?", topk: 3)
→ Returns only the passages that answer the question, not the whole page
(Jina read_url question/topk; fallback: Firecrawl scrape in "query" format)
Use it by default in READ. Read in full when you need exact quotes or tables, or the passages were not enough.
For JS-heavy pages:
If ~~scrape returns empty, try provider with waitFor/JS rendering support.
Reading Multiple URLs — ~~batch_scrape
Fallback chain:
1. Jina read_url with an array — up to 5 URLs in one call
2. Exa fetch with several URLs in one call (maxCharacters: 20000; default is 3000 per page)
3. Sequential ~~scrape per URL individually
More than 5 URLs → split into batches of 5. Only the Jina call honours question; the fallbacks return full pages, so select the passages yourself (or use the Firecrawl "query" format per URL).
Workflow: search → rank → read
1. ~~search(query) → list of URLs
2. Rank by relevance → top-5
3. ~~batch_scrape(top_5_urls) → content
Crawling a Site — ~~crawl
Fallback chain:
1. Firecrawl crawl — waits for the crawl to finish and returns the data
(check-status only to pick up a crawl started earlier or cut off)
2. Firecrawl map → list of URLs, then ~~batch_scrape(selected_urls)
With filtering:
Crawl with limit, includePaths, excludePaths to control scope.
Structured Data Extraction — ~~extract
Fallback chain:
1. Firecrawl scrape in "json" format with a prompt and schema — one URL per call
2. Firecrawl agent — when the URLs are not known or the data is spread across sites
3. ~~scrape, then extract the fields yourself
With JSON schema (known URL):
~~extract(url, prompt, schema) → structured data matching your schema
→ repeat for each URL
Unknown URLs or several sites:
1. Start the agent with a prompt, optional seed URLs, schema and a credit limit → job id
2. Poll the job every 15-30 seconds until "completed" or "failed" (1-3 minutes)
The agent spends credits — always set a limit.
Browser Sessions
For dynamic pages requiring interaction (Firecrawl only):
1. Open a page with a natural-language instruction → scrapeId
2. Follow up on the same scrapeId (navigate, click, type)
3. Stop the session
The session acts on the live site — form submissions have side effects.
PDF Processing
Text of a PDF: ~~scrape(pdf_url) — read_url and Firecrawl scrape both read PDFs.
Figures, tables, equations: PDF extraction (by arXiv id or URL) returns them as images.
Papers from the paper index: ~~academic_search read-paper step returns the passages
that answer a question.
Autonomous Research Agent
For complex multi-step research — the ~~deep_agent capability (see CONNECTORS.md):
1. Start the agent with a prompt and optional URLs → job id
2. Poll the job every 15-30 seconds until "completed" (typically 1-3 minutes)
Utility Tools
| Tool | Purpose |
|---|---|
| Screenshot | Capture a webpage for visual reports (ask for URLs, not base64 images) |
| Date detection | Detect page publication date for freshness |
Best Practices
- Always start with ~~scrape — uses fallback chain automatically
- JS-heavy pages — use provider with waitFor/JS rendering
- 3+ URLs — use ~~batch_scrape for parallel reading
- Large sites — ~~crawl with limit and depth
- PDFs — use PDF extraction for academic papers
- Structured data — ~~extract with JSON schema
- Check dates — detect page date to filter stale content
- Browser sessions — only for SPA, always stop the session after use
- Long pages — pass a
questionto read only the relevant passages
Common Errors
| Error | Fix |
|---|---|
| Empty content | Try next provider in fallback chain |
| Timeout | Try lighter alternative (map instead of crawl) |
| 403 Forbidden | Try provider with proxy/stealth support |
| PDF extraction failed | Check URL, try ~~scrape instead |
| Crawl stuck | Set limit and depth, or use map + batch_scrape |