Web Scraping

Use when "scrape this site at scale", "fetch N thousand pages", "how many workers should this scraper use", "am I hammering this server", "we're getting 429s / blocked / CAPTCHAs", "should we use a proxy", "Zyte", "ScrapingBee", "Web Unlocker", "residential proxy", "rotate proxies", "scrape with Web Unlocker", "query Bright Data", "Bright Data datasets", "Bright Data Web Archive / Wayback alternative", "FINRA BrokerCheck data", "SEC IAPD / adviserinfo data", "Investment Adviser Public Disclosure", "broker/adviser disclosure snapshots", "LinkedIn/Crunchbase/Glassdoor company or people dataset", or any use of the Bright Data API (datasets/list, Web Archive search/dump, Web Unlocker zones).

edwinhu Updated 26 repo stars

File contents

edwinhu/workflows commit ad0e173aa3

Frequently asked questions

npx get-skill add edwinhu/web-scraping