# Scraping

> Web scraping with Firecrawl and Bright Data. JavaScript rendering, anti-bot bypass, structured data extraction. Use for market research, competitor analysis, documentation. Don't use for authenticated/private content.

- Skill: `neuron-one/scraping` (Agent Skill)
- Install (CLI): `npx skillmds@latest add neuron-one/scraping`
- Raw SKILL.md: https://api.skillmd.com/api/skills/neuron-one/scraping/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: neuron-one (https://skillmd.com/u/neuron-one)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/neuron-one/scraping

---


# Web Scraping

## Tool Selection
- Firecrawl: general scraping, JS rendering (primary tool)
- Bright Data MCP: when site blocks normal scraping (anti-bot bypass)
- Direct WebFetch: simple pages without protection

## Best Practices
- Respect robots.txt
- Rate limit requests (1-2 sec between)
- Cache results locally
- Structure output as JSON/CSV

## Legal Notes
- Public data only
- Check ToS of target site
- Don't scrape personal data without consent