Botasaurus CLI

botasaurus-cli — Web Scraping CLI

Scrape web pages from the command line with Cloudflare bypass and anti-detect HTTP requests. Wraps Botasaurus (omkarcloud/botasaurus, 4.7k ⭐, MIT).

Installation

pip install botasaurus          # Core scraping engine (~14MB deps)
pip install botasaurus-cli      # CLI wrapper

First run auto-downloads @request dependencies (~14MB).

Commands

• botasaurus-cli web scrape <url> [--json] — Scrape a URL
• botasaurus-cli web check <url> [--json] — Check if URL is accessible
• botasaurus-cli self version — Show engine info

Critical Caveats & Pitfalls

1. First Run: Dependency Download

The first scrape or check call downloads the botasaurus_requests library (~14MB). Always warn the user about this before running for the first time.

Downloading @request dependencies...
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.8/13.8 MB

Takes 10-30 seconds depending on connection. Subsequent calls are instant.

2. Python ctypes Requirement

Botasaurus requires Python with ctypes support. Some Python installations (e.g., certain pyenv builds, system Python on some distros) lack _ctypes and will fail with:

ModuleNotFoundError: No module named '_ctypes'

Fix: Use a different Python version (python3.10 usually works). The CLI auto-detects the correct Python.

3. First Run Timeout

The first scrape/check can take 15-30 seconds due to dependency download + setup. Set timeout_ms high enough (120000+ recommended) when calling via supercli.

4. Cloudflare Bypass is Best-Effort

Botasaurus's @request decorator handles basic Cloudflare challenges (Connection Challenge via Google Referrer). It may NOT bypass:

• JS with Captcha challenges (need @browser with bypass_cloudflare=True)
• Complex WAF rules (Datadome, PerimeterX)
• Turnstile CAPTCHAs

If a URL fails: Suggest trying with a browser-based tool or direct curl.

5. No Browser Rendering

botasaurus-cli uses the @request decorator which sends HTTP requests, not a full browser. It will NOT execute JavaScript or render SPAs (React, Vue, etc.). For JS-heavy sites, use the full Botasaurus framework with @browser decorator.

6. Output Format

• Default: human-readable summary (URL, status, title, size)
• --json: structured JSON with all fields
• Title extraction uses BeautifulSoup (soupify) — only works for HTML pages, not JSON APIs

7. Rate Limiting & Politeness

Botasaurus does NOT add delays between requests by default. When scraping multiple URLs, the user should add their own rate limiting. Aggressive scraping may get the user's IP blocked.

8. Bot Detection

While Botasaurus is designed to bypass bot detection, results vary by target:

• ✅ Works on most Cloudflare-protected sites
• ✅ Works on sites with basic bot detection
• ⚠️ May fail on advanced anti-bot systems (DataDome, Akamai)
• ⚠️ Headless Chrome (@browser) works better but is not exposed in this CLI

9. Error Handling

If a scrape fails, common causes:

• Timeout: Site is slow or blocking. Retry with longer timeout.
• 403/503: Blocked by WAF. Try a different approach (browser-based tool).
• SSL errors: Site has certificate issues. Use --json to see raw response.
• Empty title: Page is not HTML (JSON, image, etc.) or JS-rendered.

10. API Responses

For scraping JSON APIs (REST, GraphQL), scrape returns the raw response length but can't parse it. The user should use curl or dedicated API tools for structured API responses.

Prompt Templates

• "Check if [URL] is accessible and returns 200"
• "Scrape the title and metadata from [URL]"
• "Can you access this Cloudflare-protected page: [URL]?"
• "Tell me if this URL is behind Cloudflare"
• "Extract the page content from [URL] and return JSON"

Typical Workflow

→ "Check if example.com is accessible"
← botasaurus-cli web check "https://example.com"

→ "Scrape the title from that page"
← botasaurus-cli web scrape "https://example.com"

→ "Now try this Cloudflare site: https://nopecha.com/demo/cloudflare"
← botasaurus-cli web scrape "https://nopecha.com/demo/cloudflare"
   (may fail — JS challenges require @browser mode)

Caveats Summary