# Webclaw

> Use this skill when the user wants to scrape web pages, crawl websites, or extract structured content from URLs for LLM consumption.

- Skill: `javimosch/webclaw` (Agent Skill)
- Install (CLI): `npx skillmds@latest add javimosch/webclaw`
- Raw SKILL.md: https://api.skillmd.com/api/skills/javimosch/webclaw/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: javimosch (https://skillmd.com/u/javimosch)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/javimosch/webclaw

---


# webclaw Plugin

Fast, local-first web content extraction for LLMs. webclaw scrapes single pages, crawls entire sites, and outputs clean structured data in markdown, JSON, text, or LLM-optimized formats. Zero browser overhead.

## Commands

### Extraction
- `webclaw page extract` — Extract content from a single web page
- `webclaw site crawl` — Crawl a website and extract from multiple pages
- `webclaw page diff` — Compare a page against a previous snapshot
- `webclaw self version` — Print webclaw version
- `webclaw _ _` — Passthrough to webclaw CLI

## Usage Examples
- "Extract the content from https://example.com as markdown"
- "Crawl https://docs.example.com to depth 2"
- "Scrape a page and output JSON"
- "Extract brand information from a website"

## Installation

```bash
brew tap 0xMassi/webclaw && brew install webclaw
```

## Examples

```bash
# Extract a single page as markdown
webclaw page extract https://stripe.com -f markdown

# Extract main content only
webclaw page extract https://example.com --only-main-content

# Extract with specific CSS selectors
webclaw page extract https://example.com --include "article, .content" --exclude "nav, footer"

# Extract brand assets
webclaw page extract https://github.com --brand

# Crawl a docs site to depth 2, max 50 pages
webclaw site crawl https://docs.rust-lang.org --depth 2 --max-pages 50 -f markdown

# Crawl from sitemap
webclaw site crawl https://example.com --sitemap -f json

# Discover URLs only
webclaw site crawl https://example.com --map

# Extract as LLM-optimized format
webclaw page extract https://example.com -f llm

# Compare page against a snapshot
webclaw page extract https://example.com -f json > snap.json
webclaw page diff snap.json https://example.com
```

## Key Features
- Sub-millisecond extraction with zero browser overhead
- 5 output formats: markdown, text, JSON, LLM-optimized, HTML
- CSS selector filtering (--include, --exclude)
- Auto-detect main content with --only-main-content
- Same-origin BFS crawling with depth and max-pages limits
- Sitemap-based crawling
- Brand extraction (colors, fonts, logos)
- Change tracking with snapshot diffing
- PDF and YouTube metadata extraction
- Readability scoring and noise filtering