agentskillexchange Crawlee Web Crawling and Browser Automation Library
Builds scalable web collection pipelines with Crawlee, Apify’s open-source crawling and browser automation library. Useful for request queue management, Playwright or Puppeteer crawling, structured dataset export, and resilient scraping across large sets of pages.
Installation
Use the upstream install or setup path that matches your environment:
- npx crawlee create my-crawler
- npm start
- npm install crawlee playwright
- npm install crawlee@next
Requirements and caveats from upstream:
- Do you prefer 🐍 Python instead of JavaScript? 👉 Checkout Crawlee for Python 👈.
- Crawlee requires Node.js 16 or higher.
Basic usage or getting-started notes:
We recommend visiting the Introduction tutorial in Crawlee documentation for more information.
With Crawlee CLI
The fastest way to try Crawlee out is to use the Crawlee CLI and choose the Getting started example. The CLI will install all the necessary dependencies and add boilerplate code for you to play with.
Source: https://github.com/apify/crawlee
Extracted from upstream docs: https://raw.githubusercontent.com/apify/crawlee/HEAD/README.md