How do I install the Crawl4ai Foundation skill?

Run npx skillmds@latest add ryan-brosas/crawl4ai-foundation in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Crawl4ai Foundation safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Crawl4ai Foundation?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Crawl4ai Foundation free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Crawl4ai Foundation?

ryan-brosas (@ryan-brosas) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Crawl4ai Foundation by ryan-brosas — Agent Skill

Name: Crawl4ai Foundation
Author: ryan-brosas

Crawl4ai Foundation

Use when building or porting async web crawlers - anti-bot retry/detection tiers, smart cache validation, memory-adaptive dispatch, deep-crawl traversal guards, sitemap/Common-Crawl seeding, and recycled browser fleets - capsule-v2 source maps with decisive excerpts and graph retrieval.

ryan-brosas 272c169 14 files · 52.6 KB Updated Sep 17, 2026

File contents

crawl4ai: async crawler foundations

Foundation contract

This is a cold, source-specific foundation, not an operational procedure. It is historical, revision-pinned evidence rather than timeless guidance. Current project source, tests, requirements, and runtime behavior outrank this projection and decide what ships.

Provenance and revision

Source record: crawl4ai; portable upstream identity, license, coverage caveats, and retrieval history are preserved in the index.
Recorded revision: main@7e801521428ee12509994d39151006f64055ebe3.
Full provenance, coverage caveats, boundaries, and preserved evidence: references/index.md.

Topic map

Representative topics: Fetch & resilience; Cache plane; Multi-URL orchestration; URL discovery; Browser fleet.
Scope and retrieval questions: references/index.md#use-this-for.
Complete capsule chooser: references/index.md#load-the-matching-source-dump.
Detailed grouped map: references/index.md#capsule-map.
Source limitations and portability boundaries: references/index.md#boundaries.

Retrieval

Open the index, choose one capsule matching the active question, and load only that capsule. Revalidate its cited source and revision before relying on it. Do not bulk-load the inventory, treat historical claims as current truth, or apply this foundation as a procedure.

Frequently asked questions