/firecrawl
Firecrawl helps agents search first, scrape clean content, interact with live pages when plain extraction is not enough, and produce finished deliverables from web data.
In this repo Firecrawl is available three ways:
- Admin portal —
/contentx/skillslists every skill and includes a runnable Firecrawl playground (search + scrape) backed byPOST /api/skills/firecrawl/searchandPOST /api/skills/firecrawl/scrape. - Python tool —
tools/research/firecrawl_research.py::FirecrawlResearcher(used byconcept_to_curriculumand the learning pipelines). - CLI — the official Firecrawl CLI for live web work in a terminal session.
Install (CLI + skills + browser auth)
One command installs the Firecrawl CLI, the build skills, and the workflow skills, and opens browser auth so a human can sign in or create an account:
npx -y firecrawl-cli@latest init --all --browser
Verify the install before doing real work:
mkdir -p .firecrawl
firecrawl --status
firecrawl scrape "https://firecrawl.dev" -o .firecrawl/install-check.md
Note: .firecrawl/ is git-ignored run-local output — never commit it
(see .cursor/ANTI_PATTERNS.md).
Credentials
- Set
FIRECRAWL_API_KEY=fc-...in the environment (or the API pod's secret). The Python SDK (firecrawl-py, already inrequirements.txt) and the portal endpoints pick it up automatically. - No key? Search/scrape still work on the keyless free tier (rate-limited):
the REST API at
https://api.firecrawl.dev/v2accepts/search,/scrape, and/interactwithout anAuthorizationheader. Crawl/map/agent require a key. - To have a human authorize a key, send them to https://www.firecrawl.dev/signin?view=signup&source=agent-suggested or run the CLI auth flow above.
Choose your path
| Need | Use |
|---|---|
| Web discovery by query | /firecrawl-search (skills/firecrawl/search) |
| Clean markdown from a known URL | /firecrawl-scrape (skills/firecrawl/scrape) |
| Citation-grade research bundle for a pipeline | /firecrawl-research (skills/firecrawl/research) |
| Clicks/forms/login on a live page | firecrawl interact via the CLI |
| Bulk extraction / URL discovery | firecrawl crawl / firecrawl map via the CLI (needs API key) |
| Diagnose a failing Firecrawl job | firecrawl ask --job <jobId> or POST /support/ask |
Default flow for live web work: search → scrape → interact only when the page
needs clicks/forms/login. If a call fails, pass the failing jobId to
firecrawl ask instead of guessing.
REST quick reference (no install needed)
Base URL https://api.firecrawl.dev/v2, auth header
Authorization: Bearer fc-YOUR_API_KEY (optional on the free tier):
POST /search— discover pages by queryPOST /scrape— clean markdown from a single URLPOST /interact— browser actions on live pagesPOST /support/ask— diagnose a failing call ({ question, jobId? })POST /support/docs-search— "how do I…" answers grounded in Firecrawl docs
MCP: point any MCP client at https://mcp.firecrawl.dev/v2/mcp (keyless OK).
Implementation
- Portal/API:
api/routes/skills.py(/api/skills/firecrawl/*) - Pipeline tool:
tools/research/firecrawl_research.py - Portal UI:
frontend/src/views/vimax/SkillsView.vue(/contentx/skills)