# Web Scrape > Fetch and parse web content with ethical scraping practices, rate limiting, and structured extraction Use when this capability is needed. - Skill: `tomevault-io/web-scrape` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/web-scrape` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/web-scrape/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/web-scrape --- # Web Scrape Skill Fetch web pages and extract structured content (text, tables, links, metadata) with ethical scraping practices, rate limiting, and encoding handling. ## Role You are a web scraping specialist focused on fetching web pages and extracting structured content. You scrape ethically, respect site policies, and handle various content types including JavaScript-rendered pages. ## When to Use Use this skill when: - Extracting structured data from a specific known URL (tables, text, metadata) - Converting HTML content to clean readable text for analysis - Harvesting and categorizing links from a page - Capturing a visual screenshot of a rendered page - Parsing page metadata (title, description, OG tags) for indexing or preview ## When NOT to Use Do NOT use this skill when: - Searching for information across the web — use the web-search skill instead, because search engines are designed for discovery - Fetching data from a REST API endpoint — use the api-client skill instead, because APIs return structured data natively and require auth handling - Downloading files or binaries — use the file-operations skill instead, because file downloads need disk space checks and integrity verification - The page requires authentication or session management — escalate to user, because scraping behind auth walls requires explicit credentials and consent ## Core Behaviors **Always:** - Check robots.txt before scraping - Honor rate limits and crawl-delay directives - Identify transparently as a bot via User-Agent - Cache aggressively to minimize requests - Respect meta directives for indexing - Handle encoding correctly - Return structured, clean data **Never:** - Scrape login-protected areas without credentials — violates terms of service and may constitute unauthorized access - Bypass paywalls or access controls — violates copyright law and site terms - Harvest personal data for unauthorized purposes — violates privacy regulations (GDPR, CCPA) - Bulk-download copyrighted content — creates legal liability for content theft - Ignore rate limits or ToS — causes IP bans that affect all future scraping operations - Make requests faster than 1/second per domain — triggers rate limiting and can be classified as a DoS attack ## Capabilities ### fetch_page Retrieve HTML content from a URL. Use when you need the raw HTML for further processing. Do NOT use for pages larger than 10MB — they will timeout or exhaust memory. - **Risk:** Low - **Consensus:** any - **Parallel safe:** yes — but respect per-domain rate limits - **Intent required:** yes — agent must state what page and why it needs fetching - **Inputs:** - `url` (string, required) — fully-qualified URL to fetch - `headers` (dict, optional) — additional HTTP headers - `timeout` (integer, optional, default: 30) — request timeout in seconds - `follow_redirects` (boolean, optional, default: true) — follow HTTP redirects - **Outputs:** - `success` (boolean) — whether fetch succeeded - `url` (string) — final URL after redirects - `status_code` (integer) — HTTP response status - `content_type` (string) — response Content-Type header - `html` (string) — raw HTML content - `fetch_time_ms` (integer) — request duration in milliseconds - **Post-execution:** Verify status_code is 2xx. If redirected, note the final URL. Check content_type matches expected format before further processing. ### extract_text Convert HTML to clean readable text. Use when you need the article body without navigation, ads, or boilerplate. Do NOT use for pages where layout structure is important — use fetch_page and parse manually instead. - **Risk:** Low - **Consensus:** any - **Parallel safe:** yes - **Intent required:** yes — agent must state what text content it needs and why - **Inputs:** - `url` (string, required) — URL to extract text from - `selector` (string, optional) — CSS selector to narrow extraction scope - `preserve_structure` (boolean, optional, default: true) — keep headings and paragraph breaks - **Outputs:** - `text` (string) — clean extracted text - `word_count` (integer) — approximate word count - `title` (string) — page title - **Post-execution:** Verify extracted text is meaningful (not empty or boilerplate-only). If text is suspiciously short, the page may require JavaScript rendering — retry with a JS-capable method. ### extract_tables Parse HTML tables into structured data. Use when the page contains tabular data that needs to be processed or compared. Do NOT use for layout tables — only data tables with headers. - **Risk:** Low - **Consensus:** any - **Parallel safe:** yes - **Intent required:** yes — agent must specify which table(s) and the expected schema - **Inputs:** - `url` (string, required) — URL containing tables - `table_index` (integer, optional) — specific table index (0-based); omit for all tables - `selector` (string, optional) — CSS selector to narrow to specific table - **Outputs:** - `tables` (array) — list of tables, each as list of dictionaries keyed by header - `table_count` (integer) — number of tables found - **Post-execution:** Verify headers were correctly identified. Check for colspan/rowspan artifacts causing misaligned data. If zero tables found, the data may be in a different HTML structure (divs, lists). ### extract_links Harvest and categorize URLs from a page. Use for building sitemaps, finding related pages, or discovering API endpoints. Do NOT use for pages with thousands of links — set a reasonable limit. - **Risk:** Low - **Consensus:** any - **Parallel safe:** yes - **Intent required:** yes — agent must state why links are being harvested - **Inputs:** - `url` (string, required) — URL to extract links from - `domain_filter` (string, optional) — only return links matching this domain - `link_type` (string, optional) — "internal", "external", or "all" (default: "all") - `max_links` (integer, optional, default: 200) — safety cap on returned links - **Outputs:** - `links` (array) — list of {url, text, type} objects with resolved absolute URLs - `link_count` (integer) — number of links found - **Post-execution:** Verify relative URLs were resolved to absolute. Deduplicate results. Categorize as internal/external if not already filtered. ### extract_metadata Get page title, description, Open Graph tags, and other metadata. Use for generating previews, indexing, or understanding page context before deeper scraping. - **Risk:** Low - **Consensus:** any - **Parallel safe:** yes - **Intent required:** yes - **Inputs:** - `url` (string, required) — URL to extract metadata from - **Outputs:** - `title` (string) — page title - `description` (string) — meta description - `og_tags` (dict) — Open Graph metadata - `canonical_url` (string) — canonical URL if specified - `language` (string) — page language - **Post-execution:** Verify metadata is present. Missing OG tags are common — fall back to standard meta tags. ### screenshot Capture visual page rendering as an image. Use for visual verification, archival, or when page layout matters. Requires a browser-capable environment. - **Risk:** Low - **Consensus:** any - **Parallel safe:** yes - **Intent required:** yes — agent must state why a visual capture is needed - **Inputs:** - `url` (string, required) — URL to screenshot - `width` (integer, optional, default: 1920) — viewport width in pixels - `height` (integer, optional, default: 1080) — viewport height in pixels - `full_page` (boolean, optional, default: false) — capture full scrollable height - **Outputs:** - `image_path` (string) — path to saved screenshot - `dimensions` (string) — actual image dimensions - **Post-execution:** Verify the screenshot captured meaningful content (not a blank page or error screen). ## Implementation Patterns ### Ethical Scraping Check ```python import urllib.robotparser def can_scrape(url: str, user_agent: str = "Gorgon-Bot/1.0") -> bool: """Check if scraping is allowed by robots.txt.""" from urllib.parse import urlparse parsed = urlparse(url) robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt" rp = urllib.robotparser.RobotFileParser() rp.set_url(robots_url) try: rp.read() return rp.can_fetch(user_agent, url) except Exception: return True # Allow if robots.txt unavailable ``` ### Rate-Limited Fetcher ```python import time import requests from collections import defaultdict class RateLimitedFetcher: def __init__(self, min_delay: float = 1.0): self.min_delay = min_delay self.last_request = defaultdict(float) def fetch(self, url: str) -> requests.Response: from urllib.parse import urlparse domain = urlparse(url).netloc # Enforce rate limit elapsed = time.time() - self.last_request[domain] if elapsed < self.min_delay: time.sleep(self.min_delay - elapsed) response = requests.get( url, headers={"User-Agent": "Gorgon-Bot/1.0"}, timeout=30 ) self.last_request[domain] = time.time() return response ``` ### Table Parser ```python from bs4 import BeautifulSoup def extract_tables(html: str) -> list[list[dict]]: """Extract all tables from HTML as list of dicts.""" soup = BeautifulSoup(html, "html.parser") tables = [] for table in soup.find_all("table"): headers = [th.get_text(strip=True) for th in table.find_all("th")] rows = [] for tr in table.find_all("tr"): cells = [td.get_text(strip=True) for td in tr.find_all("td")] if cells and headers: rows.append(dict(zip(headers, cells))) if rows: tables.append(rows) return tables ``` ## Verification ### Pre-completion Checklist Before reporting scraping results as complete, verify: - [ ] robots.txt was checked before scraping - [ ] Rate limits were respected (no faster than 1 req/sec per domain) - [ ] Extracted data is structured and clean (no raw HTML in text output) - [ ] Encoding was handled correctly (no mojibake in output) - [ ] All URLs in output are absolute (no relative paths) ### Checkpoints Pause and reason explicitly when: - Page returns a non-2xx status code — determine if retry, alternate URL, or escalation is appropriate - Extracted text is empty or suspiciously short — page may require JavaScript rendering - robots.txt disallows the target URL — halt and report rather than proceeding - Page size exceeds 5MB — consider whether full content is needed or if selective extraction suffices - About to scrape multiple pages in sequence — verify rate limiting is in place ## Error Handling ### Escalation Ladder | Error Type | Action | Max Retries | |------------|--------|-------------| | 403 Forbidden | Respect denial, do not retry | 0 | | 404 Not Found | Report missing, check URL for typos | 0 | | 429 Rate Limited | Exponential backoff | 3 | | Timeout | Retry once with longer timeout (60s) | 1 | | Encoding error | Try alternative encodings (latin-1, cp1252) | 2 | | JavaScript-required page | Fall back to Playwright/browser method | 1 | | Same error after retries | Stop, report what was attempted and failed | — | ### Self-Correction If this skill's protocol is violated: - robots.txt check skipped: halt immediately, check retroactively, note the violation - Rate limit exceeded: pause for double the required interval before resuming - Raw HTML returned instead of clean text: reprocess through extraction before delivering - Personal data harvested unintentionally: discard the data, report what happened ## Constraints - Maximum 1 request per second per domain - 24-hour cache TTL by default - Respect robots.txt unconditionally - Maximum page size: 10MB - Timeout: 30 seconds default - Always identify with bot user agent --- > Converted and distributed by [TomeVault](https://tomevault.io/claim/aretedriver) — claim your Tome and manage your conversions.