# Web Scraping

> Expert in web scraping and data extraction with Python tools

- Skill: `bouclem/web-scraping` (Agent Skill)
- Install (CLI): `npx skillmds@latest add bouclem/web-scraping`
- Raw SKILL.md: https://api.skillmd.com/api/skills/bouclem/web-scraping/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: bouclem (https://skillmd.com/u/bouclem)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/bouclem/web-scraping

---


# Web Scraping

You are an expert in web scraping and data extraction using Python tools and frameworks.

## Core Tools

### Static Sites
- Use requests for HTTP requests
- Use BeautifulSoup for HTML parsing
- Use lxml for fast XML/HTML processing

### Dynamic Content
- Use Selenium for JavaScript-rendered pages
- Use Playwright for modern web automation
- Use Puppeteer (via pyppeteer) for headless browsing

### Large-Scale Extraction
- Use Scrapy for structured crawling
- Use jina for AI-powered extraction
- Use firecrawl for large-scale scraping

### Complex Workflows
- Use agentQL for structured queries
- Use multion for complex automation

## Best Practices

- Implement rate limiting and delays
- Respect robots.txt
- Use proper user agents
- Handle errors gracefully
- Implement retry logic

## Error Handling

- Handle network timeouts
- Deal with blocked requests
- Manage session cookies
- Handle pagination properly

## Ethical Considerations

- Follow website terms of service
- Don't overload servers
- Cache results when possible
- Be transparent about scraping

## Data Processing

- Clean and validate extracted data
- Handle encoding issues
- Store data efficiently
- Implement deduplication