How do I install the Crawl4ai skill?

Run npx skillmds@latest add cyz26/crawl4ai in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Crawl4ai safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Crawl4ai?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Crawl4ai free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Crawl4ai?

cYz26 (@cyz26) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Crawl4ai by cyz26 — Agent Skill

Name: Crawl4ai
Author: cYz26

Crawl4ai

Use when web pages or HTTP(S) URLs should be crawled, browser-rendered, or converted into Markdown with Crawl4AI for research, source intake, summarization, review, or downstream knowledge-base workflows; use MarkItDown instead for local PDFs, Office files, archives, downloaded documents, and attachments.

cYz26 Updated Sep 21, 2026

File contents

Crawl4AI

Overview

Crawl live web pages with Crawl4AI and return reviewable Markdown. Keep this skill focused on HTTP(S) pages; use document-oriented converters for local files and downloaded artifacts.

Workflow

Confirm the input is a user-provided URL or a narrowly scoped set of URLs.
Check whether Crawl4AI is available before promising extraction.
Use scripts/crawl4ai_fetch.py for repeatable command execution and JSON reporting when a reusable artifact is useful.
Preserve the original URL, fetched Markdown, command status, and any skipped reason in the target workflow.
Use AgentKB kb-import only when the user wants the result stored in an AgentKB vault; otherwise keep the output in the current task's artifact path.

Runtime Detection

The helper resolves Crawl4AI in this order:

CRAWL4AI_CMD
AGENT_KB_CRAWL4AI_CMD
~/.codex/crawl4ai-venv/bin/crwl
~/.codex/agent-kb/crawl4ai-venv/bin/crwl
crwl on PATH

Install a dedicated runtime when Crawl4AI is missing:

/opt/homebrew/bin/python3.12 -m venv ~/.codex/crawl4ai-venv
~/.codex/crawl4ai-venv/bin/python -m pip install -U crawl4ai
~/.codex/crawl4ai-venv/bin/crawl4ai-setup
~/.codex/crawl4ai-venv/bin/crawl4ai-doctor

Do not install dependencies automatically unless the user asked for setup. Do not configure a global MCP server without separately validating Codex transport compatibility.

Commands

Check availability:

python3 scripts/crawl4ai_fetch.py --check --json

Fetch Markdown:

python3 scripts/crawl4ai_fetch.py --url https://example.com --json

Write only the content field into downstream Markdown files unless the target workflow asks for a JSON receipt.

Boundaries

Use Crawl4AI for live web pages where JavaScript rendering, cleanup, and Markdown generation are useful.
Use MarkItDown for local PDFs, Office files, archives, attachments, and downloaded documents.
Do not broaden a single URL request into site crawling unless the user gives an explicit scope and limit.
Avoid authenticated, private, or internal URLs unless the user has provided clear authorization and the output destination is appropriate.
Treat failed, empty, or unavailable Crawl4AI output as a skipped/degraded state rather than inventing content.

AgentKB Integration

When the user wants crawled pages to enter an AgentKB vault, use kb-import. AgentKB consumes the same command conventions and records raw URL metadata, extracted Markdown, source summaries, receipts, and registry entries. This skill remains the reusable Crawl4AI guidance for non-AgentKB contexts.

---name: crawl4aidescription: Use when web pages or HTTP(S) URLs should be crawled, browser-rendered, or converted into Markdown with Crawl4AI for research, source intake, summarization, review, or downstream knowledge-base workflows; use MarkItDown instead for local PDFs, Office files, archives, downloaded documents, and attachments.---# Crawl4AI## OverviewCrawl live web pages with Crawl4AI and return reviewable Markdown. Keep thisskill focused on HTTP(S) pages; use document-oriented converters for localfiles and downloaded artifacts.## Workflow1. Confirm the input is a user-provided URL or a narrowly scoped set of URLs.2. Check whether Crawl4AI is available before promising extraction.3. Use `scripts/crawl4ai_fetch.py` for repeatable command execution and JSON reporting when a reusable artifact is useful.4. Preserve the original URL, fetched Markdown, command status, and any skipped reason in the target workflow.5. Use AgentKB `kb-import` only when the user wants the result stored in an AgentKB vault; otherwise keep the output in the current task's artifact path.## Runtime DetectionThe helper resolves Crawl4AI in this order:1. `CRAWL4AI_CMD`2. `AGENT_KB_CRAWL4AI_CMD`3. `~/.codex/crawl4ai-venv/bin/crwl`4. `~/.codex/agent-kb/crawl4ai-venv/bin/crwl`5. `crwl` on `PATH`Install a dedicated runtime when Crawl4AI is missing:```bash/opt/homebrew/bin/python3.12 -m venv ~/.codex/crawl4ai-venv~/.codex/crawl4ai-venv/bin/python -m pip install -U crawl4ai~/.codex/crawl4ai-venv/bin/crawl4ai-setup~/.codex/crawl4ai-venv/bin/crawl4ai-doctor```Do not install dependencies automatically unless the user asked for setup. Donot configure a global MCP server without separately validating Codex transportcompatibility.## CommandsCheck availability:```bashpython3 scripts/crawl4ai_fetch.py --check --json```Fetch Markdown:```bashpython3 scripts/crawl4ai_fetch.py --url https://example.com --json```Write only the `content` field into downstream Markdown files unless the targetworkflow asks for a JSON receipt.## Boundaries- Use Crawl4AI for live web pages where JavaScript rendering, cleanup, and Markdown generation are useful.- Use MarkItDown for local PDFs, Office files, archives, attachments, and downloaded documents.- Do not broaden a single URL request into site crawling unless the user gives an explicit scope and limit.- Avoid authenticated, private, or internal URLs unless the user has provided clear authorization and the output destination is appropriate.- Treat failed, empty, or unavailable Crawl4AI output as a skipped/degraded state rather than inventing content.## AgentKB IntegrationWhen the user wants crawled pages to enter an AgentKB vault, use `kb-import`.AgentKB consumes the same command conventions and records raw URL metadata,extracted Markdown, source summaries, receipts, and registry entries. Thisskill remains the reusable Crawl4AI guidance for non-AgentKB contexts.

Frequently asked questions