Crawler

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations

bytesagain Updated 12 repo stars

File contents

Crawler

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only.

Commands

Command Description
intro Crawling vs scraping, robots.txt, sitemap
standards HTTP caching, structured data, meta tags
troubleshooting Anti-bot detection, JS rendering, encoding
performance Concurrency, dedup, incremental, distributed
security Legal landscape, ethical guidelines, proxies
migration BeautifulSoup to Scrapy, requests to Playwright
cheatsheet Scrapy commands, CSS/XPath, curl, user-agents
faq Legality, JS pages, blocking, storage

Output Format

All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access.


Powered by BytesAgain | bytesagain.com | hello@bytesagain.com

bytesagain/ai-skills/tree/main/crawler commit 1ec85628aa

Frequently asked questions

npx skillmds@latest add bytesagain/crawler