Scraping

Web scraping with Firecrawl and Bright Data. JavaScript rendering, anti-bot bypass, structured data extraction. Use for market research, competitor analysis, documentation. Don't use for authenticated/private content.

neuron-one 5c6c944 710 B Updated

File contents

Web Scraping

Tool Selection

  • Firecrawl: general scraping, JS rendering (primary tool)
  • Bright Data MCP: when site blocks normal scraping (anti-bot bypass)
  • Direct WebFetch: simple pages without protection

Best Practices

  • Respect robots.txt
  • Rate limit requests (1-2 sec between)
  • Cache results locally
  • Structure output as JSON/CSV

Legal Notes

  • Public data only
  • Check ToS of target site
  • Don't scrape personal data without consent

neuron-one/godmode/tree/main/skills/data/scraping commit 5c6c944f29

Frequently asked questions

npx skillmds@latest add neuron-one/scraping