# Scrapling

> Use Scrapling to scrape websites with adaptive parsing, Cloudflare bypass, and MCP support. Handles dynamic content, anti-bot detection, and provides clean HTML/JSON output.

- Skill: `dvcrn/scrapling-3` (Agent Skill)
- Install (CLI): `npx skillmds@latest add dvcrn/scrapling-3`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dvcrn/scrapling-3/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: dvcrn (https://skillmd.com/u/dvcrn)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/dvcrn/scrapling-3

---


# Scrapling Skill

Use the `scrapling` CLI to scrape websites with adaptive parsing and anti-bot bypass.

## When to Use

✅ **USE this skill when:**

- Scrape static or dynamic websites
- Bypass Cloudflare, captcha, or bot detection
- Extract structured data (HTML/JSON) from web pages
- Handle JavaScript-rendered content
- Get clean HTML without extra scripts/CSS

## When NOT to Use

❌ **DON'T use this skill when:**

- Simple HTTP requests → use `web_fetch`
- Need full browser automation → use `browser` tool
- API-based data → use direct API calls
- Local file processing → use file tools

## Setup

```bash
# Install CLI
pipx install scrapling
scrapling --version
```

## Common Commands

### Basic Scrape

```bash
# Get clean HTML
scrapling https://example.com -o html

# Get JSON structure
scrapling https://example.com -o json

# Save to file
scrapling https://example.com -o output.html
```

### With Headers/Timeouts

```bash
# Custom headers
scrapling https://example.com --headers "User-Agent: Mozilla/5.0"

# Timeout (seconds)
scrapling https://slow-site.com --timeout 30
```

### Extract Specific Elements

```bash
# XPath extraction
scrapling https://example.com -e "//div[@class='content']" -o html

# CSS selector
scrapling https://example.com -e "div.content" -o html
```

### JSON Output with Fields

```bash
# Extract title, meta description
scrapling https://example.com \
  --fields 'title,meta_description' \
  -o json
```

## MCP Integration

Scrapling supports MCP (Model Context Protocol) for AI agents:

```bash
# Start MCP server
scrapling mcp start
```

Then configure your agent to use the `scrape` tool via MCP.

## Examples

### Scrape News Article

```bash
scrapling https://example.com/news/article-123 \
  --fields 'title,author,publish_date,content' \
  -o json
```

### Extract Product Data

```bash
scrapling https://shop.example.com/products \
  -e "//div[@class='product']" \
  -o html
```

### Handle Cloudflare

```bash
# Scrapling auto-bypasses most protections
scrapling https://protected-site.com -o html
```

## Notes

- Default timeout: 10 seconds
- Auto-detects best output format (html/json/text)
- Handles dynamic content via headless browser when needed
- Rate limit friendly; add delays between requests

## JSON Output Format

```json
{
  "title": "Page Title",
  "meta_description": "Description text",
  "content": "<clean HTML>",
  "links": ["http://...", "..."],
  "images": [{"src": "...", "alt": "..."}]
}
```

Use the `scrapling` CLI to scrape websites with adaptive parsing and anti-bot bypass.

## When to Use

✅ **USE this skill when:**

- Scrape static or dynamic websites
- Bypass Cloudflare, captcha, or bot detection
- Extract structured data (HTML/JSON) from web pages
- Handle JavaScript-rendered content
- Get clean HTML without extra scripts/CSS

## When NOT to Use

❌ **DON'T use this skill when:**

- Simple HTTP requests → use `web_fetch`
- Need full browser automation → use `browser` tool
- API-based data → use direct API calls
- Local file processing → use file tools

## Setup

```bash
# Install CLI
pipx install scrapling
scrapling --version
```

## Common Commands

### Basic Scrape

```bash
# Get clean HTML
scrapling https://example.com -o html

# Get JSON structure
scrapling https://example.com -o json

# Save to file
scrapling https://example.com -o output.html
```

### With Headers/Timeouts

```bash
# Custom headers
scrapling https://example.com --headers "User-Agent: Mozilla/5.0"

# Timeout (seconds)
scrapling https://slow-site.com --timeout 30
```

### Extract Specific Elements

```bash
# XPath extraction
scrapling https://example.com -e "//div[@class='content']" -o html

# CSS selector
scrapling https://example.com -e "div.content" -o html
```

### JSON Output with Fields

```bash
# Extract title, meta description
scrapling https://example.com \
  --fields 'title,meta_description' \
  -o json
```

## MCP Integration

Scrapling supports MCP (Model Context Protocol) for AI agents:

```bash
# Start MCP server
scrapling mcp start
```

Then configure your agent to use the `scrape` tool via MCP.

## Examples

### Scrape News Article

```bash
scrapling https://example.com/news/article-123 \
  --fields 'title,author,publish_date,content' \
  -o json
```

### Extract Product Data

```bash
scrapling https://shop.example.com/products \
  -e "//div[@class='product']" \
  -o html
```

### Handle Cloudflare

```bash
# Scrapling auto-bypasses most protections
scrapling https://protected-site.com -o html
```

## Notes

- Default timeout: 10 seconds
- Auto-detects best output format (html/json/text)
- Handles dynamic content via headless browser when needed
- Rate limit friendly; add delays between requests

## JSON Output Format

```json
{
  "title": "Page Title",
  "meta_description": "Description text",
  "content": "<clean HTML>",
  "links": ["http://...", "..."],
  "images": [{"src": "...", "alt": "..."}]
}
```