Crawler
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only.
Commands
| Command |
Description |
intro |
Crawling vs scraping, robots.txt, sitemap |
standards |
HTTP caching, structured data, meta tags |
troubleshooting |
Anti-bot detection, JS rendering, encoding |
performance |
Concurrency, dedup, incremental, distributed |
security |
Legal landscape, ethical guidelines, proxies |
migration |
BeautifulSoup to Scrapy, requests to Playwright |
cheatsheet |
Scrapy commands, CSS/XPath, curl, user-agents |
faq |
Legality, JS pages, blocking, storage |
Output Format
All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access.
Powered by BytesAgain | bytesagain.com | hello@bytesagain.com
1---2name: crawler3description: Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations4---5
6# Crawler
7
8Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only.
9
10## Commands
11
12| Command | Description |
13|---------|-------------|
14| `intro` | Crawling vs scraping, robots.txt, sitemap |
15| `standards` | HTTP caching, structured data, meta tags |
16| `troubleshooting` | Anti-bot detection, JS rendering, encoding |
17| `performance` | Concurrency, dedup, incremental, distributed |
18| `security` | Legal landscape, ethical guidelines, proxies |
19| `migration` | BeautifulSoup to Scrapy, requests to Playwright |
20| `cheatsheet` | Scrapy commands, CSS/XPath, curl, user-agents |
21| `faq` | Legality, JS pages, blocking, storage |
22
23## Output Format
24
25All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access.
26
27---
28
29*Powered by BytesAgain | bytesagain.com | hello@bytesagain.com*