How do I install the Crawl4ai Selfhost skill?

Run npx skillmds@latest add sergeesteves/crawl4ai-selfhost in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Crawl4ai Selfhost safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: makes network calls, reads secrets. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Crawl4ai Selfhost?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Crawl4ai Selfhost free to use?

Yes. Installing skills from SkillMD is free. This skill is licensed under MIT.

Who published Crawl4ai Selfhost?

sergeesteves (@sergeesteves) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Crawl4ai Selfhost by sergeesteves — Agent Skill

Name: Crawl4ai Selfhost
Rating: 4.5 (4 reviews)
Author: sergeesteves

Crawl4ai Selfhost

Call a self-hosted Crawl4AI server (Docker, unclecode/crawl4ai image) to crawl pages, produce Markdown/HTML, run JavaScript, take screenshots/PDFs and do structured extraction — from Claude (MCP), n8n, or over REST/HTTP. Use whenever you have a Crawl4AI server and want to call it: endpoints, Bearer auth, the {type, params} body format, SEO patterns. For the Python SDK (AsyncWebCrawler), use the official Crawl4AI skill instead.

sergeesteves Updated Sep 22, 2026

File contents

Self-hosted Crawl4AI — server access (MCP / n8n / REST)

This skill covers calling a self-hosted Crawl4AI server (unclecode/crawl4ai image, default port 11235). For the Python SDK (AsyncWebCrawler), see the official skill: https://docs.crawl4ai.com/.

Configuration (provided by the user)

Server: {{CRAWL4AI_URL}} (e.g. http://localhost:11235).
Auth (0.9+ = secure-by-default): Authorization: Bearer <token>.
⚠️ Never hard-code the production URL or token — env vars (REST/app) or a credential (n8n).
💸 If the server routes egress through a per-GB proxy, ALL browser traffic is billed — the target page plus every third-party tracker and sub-resource it loads. Scope the proxy and cut bandwidth: references/rest-api.md → Server-side proxy & Bandwidth levers.

The 3 access paths

MCP (any MCP client, including Claude) → references/mcp.md (tools crawl / md / html / screenshot / pdf / execute_js / ask; md.f modes = fit/raw/bm25/llm).
n8n → references/n8n.md (n8n-nodes-crawl4ai-plus node, or generic HTTP Request).
REST (app / curl) → references/rest-api.md (endpoints + ⚠️ the {type, params} body gotcha).

SEO patterns

references/seo-patterns.md: BM25 Markdown per query, SEO metadata, structured extraction without an LLM, site mapping.

Rules

Check the server version / exact fields via GET {{CRAWL4AI_URL}}/schema (public, no token).
For editorial content → /md (fit or bm25 mode). For structured extraction → /crawl + extraction_strategy.
Never expose a production URL or any secret in code or logs.
Behind a metered proxy: default to no proxy; cut bandwidth (text_mode, cache_mode: "enabled", wait_until: "domcontentloaded") — put each lever in the right config class (browser_config vs crawler_config), and use NO_PROXY for trackers; proxy only as a fallback on a real block — cf. rest-api.md & n8n.md.

---name: crawl4ai-selfhostdescription: Call a self-hosted Crawl4AI server (Docker, unclecode/crawl4ai image) to crawl pages, produce Markdown/HTML, run JavaScript, take screenshots/PDFs and do structured extraction — from Claude (MCP), n8n, or over REST/HTTP. Use whenever you have a Crawl4AI server and want to call it: endpoints, Bearer auth, the {type, params} body format, SEO patterns. For the Python SDK (AsyncWebCrawler), use the official Crawl4AI skill instead.license: MIT---# Self-hosted Crawl4AI — server access (MCP / n8n / REST)This skill covers **calling a self-hosted Crawl4AI server** (`unclecode/crawl4ai` image, default port`11235`). For the **Python SDK** (`AsyncWebCrawler`), see the official skill:<https://docs.crawl4ai.com/>.## Configuration (provided by the user)- **Server**: `{{CRAWL4AI_URL}}` (e.g. `http://localhost:11235`).- **Auth** (0.9+ = secure-by-default): `Authorization: Bearer <token>`.- ⚠️ Never hard-code the production URL or token — env vars (REST/app) or a credential (n8n).- 💸 **If the server routes egress through a per-GB proxy, ALL browser traffic is billed** — the target page *plus* every third-party tracker and sub-resource it loads. Scope the proxy and cut bandwidth: [`references/rest-api.md`](references/rest-api.md) → *Server-side proxy* & *Bandwidth levers*.## The 3 access paths- **MCP** (any MCP client, including Claude) → [`references/mcp.md`](references/mcp.md) (tools `crawl` / `md` / `html` / `screenshot` / `pdf` / `execute_js` / `ask`; `md.f` modes = fit/raw/bm25/llm).- **n8n** → [`references/n8n.md`](references/n8n.md) (`n8n-nodes-crawl4ai-plus` node, or generic HTTP Request).- **REST** (app / curl) → [`references/rest-api.md`](references/rest-api.md) (endpoints + ⚠️ **the `{type, params}` body gotcha**).## SEO patterns[`references/seo-patterns.md`](references/seo-patterns.md): BM25 Markdown per query, SEO metadata,structured extraction **without an LLM**, site mapping.## Rules1. Check the server version / exact fields via `GET {{CRAWL4AI_URL}}/schema` (public, no token).2. For editorial content → `/md` (`fit` or `bm25` mode). For structured extraction → `/crawl` + `extraction_strategy`.3. Never expose a production URL or any secret in code or logs.4. Behind a metered proxy: default to **no proxy**; cut bandwidth (`text_mode`, `cache_mode: "enabled"`, `wait_until: "domcontentloaded"`) — **put each lever in the right config class** (`browser_config` vs `crawler_config`), and use `NO_PROXY` for trackers; proxy only as a fallback on a real block — cf. `rest-api.md` & `n8n.md`.

Frequently asked questions