Crawl4AI

Free open-source multi-page website crawler — self-hosted Python crawler that outputs LLM-ready Markdown with JS rendering. USE WHEN crawl website free, crawl site, crawl pages free, free crawler, open source crawler, crawl4ai, crawl without cost, self-hosted crawl, map website, extract all pages.

mj-deving 83f52cf 2 files · 3.5 KB Updated

File contents

Customization

Before executing, check for user customizations at: ${PAI_USER_DIR}/SKILLCUSTOMIZATIONS/Crawl4AI/

Crawl4AI — Free Self-Hosted Website Crawler

Open-source Python crawler (58K+ GitHub stars). Crawls entire websites, outputs clean Markdown, handles JavaScript. Completely free, runs locally.

When to Use

  • Crawl multiple pages of a website (free alternative to BrightData Crawl)
  • Extract all pages under a URL path
  • Build a sitemap + content index
  • When BrightData costs are not justified

When NOT to Use

  • Single page scraping (→ Jina Reader, faster)
  • Anti-bot / Cloudflare bypass needed (→ BrightData)
  • Social media platforms (→ Apify)
  • High-volume at scale (→ Spider.cloud)

Workflow Routing

Request Pattern Route To
Crawl website, crawl all pages, map site, extract site Workflows/Crawl.md

Prerequisites

  • Python 3.12 ✅
  • crawl4ai package (pip install)
  • Playwright browsers (for JS rendering)

Capabilities

  • JS rendering: Yes (via Playwright)
  • Markdown output: Yes (native)
  • Multi-page crawling: Yes
  • AI extraction: Yes (with own LLM)
  • Cost: $0 (self-hosted)
  • Rate limit: Only your machine's capacity

mj-deving/pai-skills/tree/main/skills/Scraping/Crawl4AI commit 83f52cf9e1

Frequently asked questions

npx skillmds@latest add mj-deving/crawl4ai