# Web Scraping

> Web scraping tools for fetching and extracting data from web pages Use when this capability is needed.

- Skill: `tomevault-io/web-scraping-5` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/web-scraping-5`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/web-scraping-5/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/web-scraping-5

---


## Web Scraping

You have web scraping tools for fetching and extracting data from web pages:

**Single page:**
- `scrape_url` — fetch a URL and get cleaned text content + metadata (title, description, link count)
  - Use format="text" (default) for most tasks — strips all HTML
  - Use format="markdown" to preserve headings, links, lists, bold/italic
  - Use format="html" only when you need raw HTML

**Link discovery:**
- `extract_links` — fetch a page and extract all links with text and type (internal/external)
  - Use the `pattern` parameter to filter by regex (e.g. `"\\.pdf$"` for PDF links)
  - Links are deduplicated and resolved to absolute URLs

**Multi-page research:**
- `scrape_multiple` — fetch up to 10 URLs in parallel for comparison/research
  - One failure doesn't block others (uses Promise.allSettled)

**Best practices:**
- Prefer "text" format for content extraction, "markdown" for preserving structure
- Don't scrape the same domain more than 5 times per minute
- Combine with `store_deliverable` to save scraped content as job evidence
- For very large pages, the content is limited to 5MB

---
> Converted and distributed by [TomeVault](https://tomevault.io/claim/xprnetwork) — claim your Tome and manage your conversions.
<!-- tomevault:4.0:skill_md:2026-04-13 -->