# Crawlee Skill

> A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

- Skill: `gdm257/crawlee-skill` (Agent Skill)
- Install (CLI): `npx skillmds@latest add gdm257/crawlee-skill`
- Raw SKILL.md: https://api.skillmd.com/api/skills/gdm257/crawlee-skill/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: gdm257 (https://skillmd.com/u/gdm257)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/gdm257/crawlee-skill

---


# Crawlee OpenCode Skill

Crawlee is a scalable web crawling and scraping library for Node.js and TypeScript. It helps you build reliable crawlers that appear human-like and fly under the radar of modern bot protections even with default configuration.

## Quick Start

### Prerequisites
- Node.js 16 or higher

### With Crawlee CLI (Recommended)
```bash
npx crawlee create my-crawler
cd my-crawler
npm start
```

### Manual Installation
```bash
npm install crawlee playwright
```

```typescript
import { PlaywrightCrawler, Dataset } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks, log }) {
        const title = await page.title();
        log.info(`Title of ${request.loadedUrl} is '${title}'`);

        await Dataset.pushData({ title, url: request.loadedUrl });
        await enqueueLinks();
    },
});

await crawler.run(['https://crawlee.dev']);
```

## Overview

Crawlee covers your crawling and scraping end-to-end and provides tools to:
- Crawl the web for links
- Scrape data from websites
- Store extracted data to disk or cloud
- Configure behavior to suit your project's needs

### Key Features
- Single interface for HTTP and headless browser crawling
- Persistent queue for URLs to crawl (breadth & depth first)
- Pluggable storage of both tabular data and files
- Automatic scaling with available system resources
- Integrated proxy rotation and session management
- Lifecycles customizable with hooks
- CLI to bootstrap your projects
- Configurable routing, error handling and retries
- Dockerfiles ready to deploy
- Written in TypeScript with generics

## Installation

### Using CLI
The fastest way to try Crawlee is using the CLI:

```bash
npx crawlee create my-crawler
cd my-crawler
npm start
```

The CLI will install all necessary dependencies and add boilerplate code for you.

### Manual Installation
If adding Crawlee to your own project:

```bash
npm install crawlee playwright
```

Note: Playwright is not bundled with Crawlee to reduce install size. You can also use Puppeteer:

```bash
npm install crawlee puppeteer
```

### Installing Pre-release Versions
For testing new features:

```bash
npm install crawlee@next
```

If using Apify SDK, specify dependency overrides:

```json
{
    "overrides": {
        "apify": {
            "@crawlee/core": "$crawlee",
            "@crawlee/types": "$crawlee",
            "@crawlee/utils": "$crawlee"
        }
    }
}
```

## Usage

### Basic Crawler

```typescript
import { PlaywrightCrawler, Dataset } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ page, request, log }) {
        const title = await page.title();
        log.info(`Crawled: ${title}`);
        await Dataset.pushData({ title, url: request.loadedUrl });
    },
});

await crawler.run(['https://example.com']);
```

### HTTP Crawler

For simple HTTP requests without browser:

```typescript
import { HttpCrawler, Dataset } from 'crawlee';

const crawler = new HttpCrawler({
    async requestHandler({ request, body }) {
        // body is the HTML content
        await Dataset.pushData({ url: request.loadedUrl, html: body });
    },
});

await crawler.run(['https://example.com']);
```

### Cheerio Crawler

Fast jQuery-like HTML parsing:

```typescript
import { CheerioCrawler, Dataset } from 'crawlee';

const crawler = new CheerioCrawler({
    async requestHandler({ request, $, log }) {
        const title = $('title').text();
        log.info(`Title: ${title}`);
        await Dataset.pushData({ title, url: request.loadedUrl });
    },
});

await crawler.run(['https://example.com']);
```

### Enqueue Links

Crawl multiple pages automatically:

```typescript
const crawler = new PlaywrightCrawler({
    async requestHandler({ enqueueLinks }) {
        // Extract and enqueue all links from current page
        await enqueueLinks();
    },
});
```

### Custom Link Selection

```typescript
await enqueueLinks({
    selector: 'a.product-link',
    baseUrl: 'https://example.com',
});
```

### Adding URLs Manually

```typescript
// Add single URL
await crawler.addRequests(['https://example.com/page-1']);

// Add multiple URLs
await crawler.addRequests([
    'https://example.com/page-1',
    'https://example.com/page-2',
]);
```

## API Reference

### Main Crawler Classes

#### PlaywrightCrawler
Headless browser crawling using Playwright.

```typescript
import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    headless: true,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 10,
    },
    requestHandler: async ({ page, request }) => {
        // Your scraping logic
    },
});
```

#### PuppeteerCrawler
Headless browser crawling using Puppeteer.

```typescript
import { PuppeteerCrawler } from 'crawlee';

const crawler = new PuppeteerCrawler({
    headless: true,
    requestHandler: async ({ page, request }) => {
        // Your scraping logic
    },
});
```

#### CheerioCrawler
Fast HTML parsing without browser.

```typescript
import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
    requestHandler: async ({ $, request }) => {
        // $ is Cheerio instance
        $('a').each((i, el) => {
            console.log($(el).text());
        });
    },
});
```

#### HttpCrawler
Simple HTTP requests with fast HTML parsing.

```typescript
import { HttpCrawler } from 'crawlee';

const crawler = new HttpCrawler({
    requestHandler: async ({ body, request }) => {
        // body is raw HTML
    },
});
```

#### JSDOMCrawler
Browser-like environment using JSDOM.

```typescript
import { JSDOMCrawler } from 'crawlee';

const crawler = new JSDOMCrawler({
    requestHandler: async ({ window, document }) => {
        const title = document.title;
    },
});
```

### Storage Classes

#### Dataset
Store scraped data:

```typescript
import { Dataset } from 'crawlee';

// Push data to default dataset
await Dataset.pushData({ name: 'Product A', price: 99 });

// Export to file
await Dataset.exportToCSV('output.csv');
```

#### KeyValueStore
Store key-value pairs:

```typescript
import { KeyValueStore } from 'crawlee';

await KeyValueStore.setValue('state', { page: 1 });
const state = await KeyValueStore.getValue('state');
```

#### RequestQueue
Manage crawling queue:

```typescript
import { RequestQueue } from 'crawlee';

const queue = await RequestQueue.open();
await queue.addRequest({ url: 'https://example.com' });
const request = await queue.fetchNextRequest();
```

## Configuration

### Basic Configuration

```typescript
const crawler = new PlaywrightCrawler({
    // Concurrency
    maxConcurrency: 10,

    // Retries
    maxRequestRetries: 3,

    // Request timeout
    requestHandlerTimeoutSecs: 30,

    // Navigation timeout
    navigationTimeoutSecs: 30,
});
```

### Proxy Configuration

```typescript
const crawler = new PlaywrightCrawler({
    proxyConfiguration: new ProxyConfiguration({
        proxyUrls: [
            'http://proxy1.com:8000',
            'http://proxy2.com:8000',
        ],
    }),
});
```

### Session Configuration

```typescript
const crawler = new PlaywrightCrawler({
    sessionPoolOptions: {
        maxPoolSize: 100,
        sessionOptions: {
            maxUsageCount: 10,
        },
    },
});
```

### HTTP/2 Configuration

```typescript
const crawler = new HttpCrawler({
    http2: true,
    // Additional HTTP2 options
});
```

### Request Handler Options

The request handler receives these parameters:

- `request` - Request information (URL, headers, userData)
- `page` - Browser page instance (Puppeteer/Playwright)
- `$` - Cheerio instance (CheerioCrawler)
- `body` - HTML content (HttpCrawler)
- `window` - JSDOM window (JSDOMCrawler)
- `enqueueLinks` - Function to add links to queue
- `log` - Logger instance
- `sendRequest` - Make additional HTTP requests

## Development

### Project Structure
```
my-crawler/
├── package.json
├── tsconfig.json
└── src/
    └── main.ts
```

### Running Tests

```bash
# Run all tests
npm test

# Run tests in watch mode
npm run test:watch

# Run e2e tests
npm run test:e2e
```

### Building

```bash
# Build project
npm run build

# Build for production
npm run ci:build
```

### Linting

```bash
# Run linter
npm run lint

# Fix linting issues
npm run lint:fix
```

### Formatting

```bash
# Format code
npm run format

# Check formatting
npm run format:check
```

### Docker Deployment

Create Dockerfile:

```dockerfile
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
CMD ["npm", "start"]
```

## Troubleshooting

### Bot Protection
Crawlee includes anti-blocking features:

```typescript
const crawler = new PlaywrightCrawler({
    // Browser-like headers are automatic
    // TLS fingerprinting is automatic
    // Human-like behavior is automatic
});
```

### Rate Limiting
Use session and request delay:

```typescript
const crawler = new PlaywrightCrawler({
    navigationTimeoutSecs: 30,
    maxRequestRetries: 3,
    sessionPoolOptions: {
        maxPoolSize: 50,
    },
});
```

### Memory Issues
Reduce concurrency:

```typescript
const crawler = new PlaywrightCrawler({
    maxConcurrency: 5,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 5,
    },
});
```

### Timeout Errors
Increase timeout values:

```typescript
const crawler = new PlaywrightCrawler({
    requestHandlerTimeoutSecs: 60,
    navigationTimeoutSecs: 60,
});
```

### Storage Location
Default storage is `./storage` directory. Change via:

```typescript
import { Configuration } from 'crawlee';

Configuration.set('storageDir', './my-storage');
```

## Resources

### Official Resources
- Documentation: https://crawlee.dev
- GitHub: https://github.com/apify/crawlee
- NPM: https://www.npmjs.com/package/crawlee
- Discord: https://discord.gg/jyEM2PRvMU
- Stack Overflow: https://stackoverflow.com/questions/tagged/apify

### Related Tools
- Apify Platform: https://apify.com
- Apify SDK: https://sdk.apify.com
- Crawlee for Python: https://github.com/apify/crawlee-python

### Examples Repository
Check out the `docs/examples/` directory in the GitHub repo for code examples covering:
- Basic crawling
- Multiple URL crawling
- Link extraction
- File downloads
- Forms submission
- Sitemap crawling
- And more...

## Best Practices

1. **Start Simple**: Begin with HTTPCrawler or CheerioCrawler for faster scraping
2. **Use Session Management**: Enable sessions to avoid IP blocking
3. **Set Timeouts**: Configure appropriate timeouts for target sites
4. **Handle Errors**: Implement proper error handling and retries
5. **Respect Robots**: Follow website robots.txt and rate limits
6. **Use Proxies**: Rotate proxies for large-scale scraping
7. **Monitor Resources**: Track memory and CPU usage
8. **Clean Data**: Validate and clean scraped data before storage

## License

Apache License 2.0 - See [LICENSE.md](https://github.com/apify/crawlee/blob/master/LICENSE.md) for details.