# Web Anti Crawl Bypass

> Standard operating procedure for bypassing common web anti-crawl mechanisms (Cloudflare, rate limiting) using ethical and compliant methods. Rotates headers, respects robots.txt, and uses exponential backoff.

- Skill: `jadegate/web-anti-crawl-bypass` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add jadegate/web-anti-crawl-bypass`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jadegate/web-anti-crawl-bypass/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: jadegate (https://skillmd.com/u/jadegate)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jadegate/web-anti-crawl-bypass

---


# Web Anti-Crawl Bypass SOP

Standard operating procedure for bypassing common web anti-crawl mechanisms (Cloudflare, rate limiting) using ethical and compliant methods. Rotates headers, respects robots.txt, and uses exponential backoff.

## When to Use

- User wants to 403
- User wants to 429
- User wants to 503
- Keywords: web, anti-crawl, scraping, cloudflare, http

## Required Parameters

- **target_url** (string): The URL to access that is being blocked by anti-crawl

## Optional Parameters

- **max_retries** (number): Maximum number of retry attempts (default: 5)
- **initial_delay_ms** (number): Initial delay before first retry in milliseconds (default: 1000)

## Output

- **status_code** (number): Final HTTP status code
- **body** (string): Response body content
- **retries_used** (number): Number of retries performed

## Security

- Sandbox level: strict
- Max execution time: 60000ms
- Network whitelist: *
- JadeGate verified: Yes (5-layer security check passed)

## Execution Flow

Entry: `check_robots` -> 10 steps -> Exit: `return_success`, `return_retry_result`, `abort_blocked`