# Firecrawl

> Scrape, crawl, map, search, or extract structured data from web pages using the local self-hosted Firecrawl API at http://localhost:3022. Use when the user asks to scrape a URL, crawl a site, extract structured data from web pages, map a sitemap, get markdown from a webpage, or fetch web content for an AI agent. Auto-trigger on phrases like "scrape this", "crawl that site", "get markdown of", "extract data from", "firecrawl".

- Skill: `maybackcompany/firecrawl` (Agent Skill)
- Install (CLI): `npx skillmds@latest add maybackcompany/firecrawl`
- Raw SKILL.md: https://api.skillmd.com/api/skills/maybackcompany/firecrawl/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: MaybackCompany (https://skillmd.com/u/maybackcompany)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/maybackcompany/firecrawl

---


# Firecrawl (local self-host)

Local Firecrawl runs at `http://localhost:3022`. No API key needed — auth disabled in dev.

## Preflight

Before first call in a session, verify it's up:

```bash
curl -sS http://localhost:3022/ -m 3
```

If connection refused, start services:

```bash
brew services start redis postgresql@16 rabbitmq
cd ~/Desktop/01_Projects/firecrawl/apps/playwright-service-ts && pnpm dev > /tmp/firecrawl-playwright.log 2>&1 &
cd ~/Desktop/01_Projects/firecrawl/apps/api && pnpm dev > /tmp/firecrawl-api.log 2>&1 &
```

Wait until `curl http://localhost:3022/` returns `{"message":"Firecrawl API",...}`.

## Endpoints

### Single page → markdown
```bash
curl -sS -X POST http://localhost:3022/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"<URL>","formats":["markdown"]}'
```

Formats: `markdown`, `html`, `rawHtml`, `links`, `screenshot`, `screenshot@fullPage`, `json` (with schema).

### Structured extract (LLM schema)
Needs `OPENAI_API_KEY` in `apps/api/.env`.
```bash
curl -sS -X POST http://localhost:3022/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"<URL>","formats":["json"],"jsonOptions":{"schema":{...}}}'
```

### Crawl whole site (async)
```bash
# Start
curl -sS -X POST http://localhost:3022/v1/crawl \
  -H 'Content-Type: application/json' \
  -d '{"url":"<URL>","limit":50,"scrapeOptions":{"formats":["markdown"]}}'
# → returns {id, url: status URL}

# Poll
curl -sS http://localhost:3022/v1/crawl/<id>
```

### Sitemap → URL list
```bash
curl -sS -X POST http://localhost:3022/v1/map \
  -H 'Content-Type: application/json' \
  -d '{"url":"<URL>"}'
```

### Search
```bash
curl -sS -X POST http://localhost:3022/v1/search \
  -H 'Content-Type: application/json' \
  -d '{"query":"<text>","limit":10}'
```

## SDK (JS)
```bash
pnpm add @mendable/firecrawl-js
```
```ts
import FirecrawlApp from '@mendable/firecrawl-js'
const app = new FirecrawlApp({ apiUrl: 'http://localhost:3022', apiKey: 'fc-dev' })
const res = await app.scrapeUrl('https://example.com', { formats: ['markdown'] })
```

## SDK (Python)
```bash
pip install firecrawl-py
```
```python
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_url='http://localhost:3022', api_key='fc-dev')
res = app.scrape_url('https://example.com', params={'formats': ['markdown']})
```

## Limits (self-host)
- No Fire-engine → blocked sites may fail (no IP rotation, no advanced anti-bot)
- pg_cron not installed → nuq cleanup jobs skipped (dev only, not prod)
- No Supabase → no usage logging, no per-key rate limits

## Full reference
See [firecrawl_self_host memory](~/.Codex/projects/-Users-kishi/memory/firecrawl_self_host.md) for ports, services, restart procedure.

## How to invoke

- `/firecrawl` — apply this skill, await URL/task
- `/firecrawl scrape <URL>` — run scrape, return markdown
- `/firecrawl crawl <URL> [limit]` — crawl site, poll until done
- `/firecrawl extract <URL> <schema-description>` — build schema + run JSON extract
- `/firecrawl map <URL>` — return URL list