# Crawl4ai Foundation

> Use when building or porting async web crawlers - anti-bot retry/detection tiers, smart cache validation, memory-adaptive dispatch, deep-crawl traversal guards, sitemap/Common-Crawl seeding, and recycled browser fleets - capsule-v2 source maps with decisive excerpts and graph retrieval.

- Skill: `ryan-brosas/crawl4ai-foundation` (Agent Skill, multi-file: 14 files)
- Install (CLI): `npx skillmds@latest add ryan-brosas/crawl4ai-foundation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ryan-brosas/crawl4ai-foundation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ryan-brosas (https://skillmd.com/u/ryan-brosas)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ryan-brosas/crawl4ai-foundation

---

# crawl4ai: async crawler foundations

## Foundation contract

This is a cold, source-specific foundation, not an operational procedure. It is
historical, revision-pinned evidence rather than timeless guidance. Current
project source, tests, requirements, and runtime behavior outrank this
projection and decide what ships.

## Provenance and revision

- Source record: `crawl4ai`; portable upstream identity, license, coverage caveats, and retrieval history are preserved in the index.
- Recorded revision: `main@7e801521428ee12509994d39151006f64055ebe3`.
- Full provenance, coverage caveats, boundaries, and preserved evidence:
  `references/index.md`.

## Topic map

- Representative topics: Fetch & resilience; Cache plane; Multi-URL
  orchestration; URL discovery; Browser fleet.
- Scope and retrieval questions: `references/index.md#use-this-for`.
- Complete capsule chooser: `references/index.md#load-the-matching-source-dump`.
- Detailed grouped map: `references/index.md#capsule-map`.
- Source limitations and portability boundaries: `references/index.md#boundaries`.

## Retrieval

Open the index, choose one capsule matching the active question, and load only
that capsule. Revalidate its cited source and revision before relying on it. Do
not bulk-load the inventory, treat historical claims as current truth, or apply
this foundation as a procedure.