# Firecrawl Scraper

> Web scraping, crawling, and search via Firecrawl API. Converts web pages to clean markdown/JSON optimized for AI consumption. Use when you need to extract content from websites, crawl entire sites, or search and scrape web results. Handles JavaScript-heavy sites, dynamic content, and provides structured output. 96% web coverage including modern SPAs.

- Skill: `dvcrn/firecrawl-scraper` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add dvcrn/firecrawl-scraper`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dvcrn/firecrawl-scraper/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: dvcrn (https://skillmd.com/u/dvcrn)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/dvcrn/firecrawl-scraper

---


# Firecrawl Scraper

Professional web scraping powered by Firecrawl API. Converts websites to clean, AI-ready markdown or structured JSON.

## When to Use

- Extract content from web pages for analysis
- Scrape documentation sites or knowledge bases
- Crawl entire websites systematically
- Search the web and get scraped content
- Parse JavaScript-heavy or dynamic sites
- Convert HTML to clean markdown for LLM processing
- Competitive research or content aggregation

## Quick Start

**Scrape a single page:**
```bash
python3 scripts/scrape.py https://example.com
```

**Crawl a website:**
```bash
python3 scripts/scrape.py --crawl https://docs.example.com --depth 2 --limit 10
```

**Search and scrape:**
```bash
python3 scripts/scrape.py --search "AI agent frameworks" --limit 5
```

**Check crawl status:**
```bash
python3 scripts/scrape.py --crawl-status abc123
```

## Commands

### Scrape (Single Page)

Extract content from a single URL:

```bash
python3 scripts/scrape.py <url> [options]
```

**Options:**
- `--formats markdown,html,screenshot` — Output formats (default: markdown)
- `--full` — Include full page (no main content extraction)
- `--json` — Output raw JSON response

**Examples:**
```bash
# Basic scrape
python3 scripts/scrape.py https://docs.example.com

# Get HTML and markdown
python3 scripts/scrape.py https://site.com --formats markdown,html

# Full page (no content filtering)
python3 scripts/scrape.py https://site.com --full

# JSON output
python3 scripts/scrape.py https://site.com --json
```

### Crawl (Entire Website)

Systematically crawl and scrape multiple pages:

```bash
python3 scripts/scrape.py --crawl <url> [options]
```

**Options:**
- `--depth N` — Maximum crawl depth (default: 2)
- `--limit N` — Maximum pages to crawl (default: 10)
- `--json` — Output raw JSON response

**Examples:**
```bash
# Basic crawl
python3 scripts/scrape.py --crawl https://docs.site.com

# Deep crawl with limit
python3 scripts/scrape.py --crawl https://blog.com --depth 3 --limit 50

# Shallow crawl
python3 scripts/scrape.py --crawl https://site.com --depth 1 --limit 5
```

**Note:** Crawl returns a job ID. Use `--crawl-status` to check progress and retrieve results.

### Search (Web Search + Scrape)

Search the web and get scraped content from results:

```bash
python3 scripts/scrape.py --search <query> [options]
```

**Options:**
- `--limit N` — Number of results (default: 5)
- `--json` — Output raw JSON response

**Examples:**
```bash
# Search and scrape
python3 scripts/scrape.py --search "WordPress security best practices"

# More results
python3 scripts/scrape.py --search "AI agents 2026" --limit 10

# JSON output
python3 scripts/scrape.py --search "casino bonuses" --json
```

### Crawl Status

Check status of a crawl job:

```bash
python3 scripts/scrape.py --crawl-status <job-id>
```

Returns JSON with:
- Status: `scraping`, `completed`, `failed`
- Progress: Pages scraped
- Data: Scraped content (when completed)
- Credits used

## Output Formats

**Markdown** (default): Clean, LLM-ready text with preserved structure
- Headings, links, lists, code blocks maintained
- No HTML noise or styling artifacts
- Perfect for RAG, summarization, analysis

**HTML**: Full HTML source (useful for parsing specific elements)

**Screenshot**: Base64-encoded PNG of rendered page

**JSON**: Structured data extraction (custom schemas supported)

## Features

**Smart Content Extraction:**
- Automatically identifies main content
- Removes navigation, ads, footers
- Preserves document structure

**JavaScript Support:**
- Handles SPAs (React, Vue, Angular)
- Waits for dynamic content to load
- 96% web coverage

**Anti-Bot Handling:**
- Proxy management built-in
- Rate limiting handled automatically
- CAPTCHA avoidance

**Caching:**
- Smart caching reduces credits
- Configurable cache behavior

## API Key Setup

The script looks for the Firecrawl API key in:
1. `workspace/secrets/firecrawl_api_key` (OpenClaw workspace)
2. `secrets/firecrawl_api_key` (relative to current directory)
3. `FIRECRAWL_API_KEY` environment variable

Current key is stored at: `workspace/secrets/firecrawl_api_key`

## Credits & Pricing

- **Scrape**: 1 credit per page
- **Crawl**: 1 credit per page crawled
- **Search**: 1 credit per result scraped
- **Screenshot**: +1 credit
- **Advanced features**: May use additional credits

Free tier: 500 credits  
Paid plans: Starting at $16/month (3,000 credits)

## Use Cases

**Documentation Extraction:**
```bash
python3 scripts/scrape.py --crawl https://docs.framework.com --depth 2 --limit 50
```

**Competitive Research:**
```bash
python3 scripts/scrape.py --search "top casino affiliate sites" --limit 10
```

**Content Migration:**
```bash
python3 scripts/scrape.py https://old-site.com/page1 --formats markdown
```

**News Monitoring:**
```bash
python3 scripts/scrape.py --search "WordPress security updates" --limit 5
```

**Blog Scraping:**
```bash
python3 scripts/scrape.py --crawl https://blog.site.com --depth 1 --limit 20
```

## Tips

- Start with low `--limit` values to test
- Use `--depth 1` for blog homepages (gets all posts)
- `--depth 2-3` for documentation sites
- `--search` is faster than manual crawling for research
- Check `--crawl-status` regularly for long crawls
- Use `--json` for programmatic processing
- Markdown format is best for LLM consumption

## Comparison to Other Tools

**vs web_fetch tool:**
- Firecrawl: Better JS support, cleaner output, handles complex sites
- web_fetch: Faster, simpler, no API credits needed
- Use Firecrawl for: Modern sites, heavy JS, need high-quality markdown
- Use web_fetch for: Simple pages, quick checks, no credit usage

**vs browser tool:**
- Firecrawl: Optimized for scraping, structured output, no browser management
- browser: Full control, visual interaction, debugging
- Use Firecrawl for: Content extraction at scale
- Use browser for: Interactive tasks, testing, visual verification