How do I install the Web Scraping skill?

Run npx skillmds@latest add abed-zahed/web-scraping in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Web Scraping safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Web Scraping?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Web Scraping free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Web Scraping?

Abed-Zahed (@abed-zahed) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Web Scraping by abed-zahed — Agent Skill

Name: Web Scraping
Author: Abed-Zahed

File contents

/web-scraping

When to use

Searching for current information, research papers, job postings, news
Scraping a specific URL (LinkedIn, job boards, company sites, articles)
Gathering structured data from one or many pages
Any task where you'd normally say "I don't have current information on this"

Exa vs Firecrawl — decision table

Use Exa when...	Use Firecrawl when...
Finding relevant pages on a topic	You have a specific URL to scrape
Semantic/conceptual search ("best RF engineers")	The page is JS-rendered (React/Angular apps)
Discovering what exists across the web	You need clean structured data from a page
Research across many sources	Crawling multiple pages of one site
Finding people, companies, recent events	Extracting structured JSON from a page

Available tools

Exa tools:

web_search_exa — semantic search, returns clean content ready to use
web_fetch_exa — fetch full content of a known URL
web_search_advanced_exa — filtered search (date range, domain, content type, category)

Firecrawl tools:

firecrawl_scrape — extract clean content from a single URL (handles JS)
firecrawl_search — web search with content extraction
firecrawl_crawl — multi-page extraction from an entire site
firecrawl_extract — structured LLM-powered data extraction from a URL
firecrawl_batch_scrape — process multiple URLs efficiently
firecrawl_agent — autonomous web research agent for complex tasks

Workflow

Discover — use web_search_exa to find relevant pages semantically
Fetch — use firecrawl_scrape on specific URLs that need JS rendering or clean extraction
Structure — use firecrawl_extract if you need JSON output from a page
Bulk — use firecrawl_batch_scrape or firecrawl_crawl for multi-page tasks

Examples

"Find top fitness researchers last 3 years" → web_search_advanced_exa with date filter
"Scrape this LinkedIn job posting [URL]" → firecrawl_scrape
"Get all blog posts from firecrawl.dev" → firecrawl_crawl
"Extract structured job data from this page" → firecrawl_extract
"Who is writing about RF antenna design right now?" → web_search_exa

---name: web-scrapingdescription: Semantic web search via Exa and JS-heavy page scraping via Firecrawl. Use proactively whenever research, current information, or URL content is needed.---# /web-scraping## When to use- Searching for current information, research papers, job postings, news- Scraping a specific URL (LinkedIn, job boards, company sites, articles)- Gathering structured data from one or many pages- Any task where you'd normally say "I don't have current information on this"## Exa vs Firecrawl — decision table| Use Exa when... | Use Firecrawl when... ||---|---|| Finding relevant pages on a topic | You have a specific URL to scrape || Semantic/conceptual search ("best RF engineers") | The page is JS-rendered (React/Angular apps) || Discovering what exists across the web | You need clean structured data from a page || Research across many sources | Crawling multiple pages of one site || Finding people, companies, recent events | Extracting structured JSON from a page |## Available tools**Exa tools:**- `web_search_exa` — semantic search, returns clean content ready to use- `web_fetch_exa` — fetch full content of a known URL- `web_search_advanced_exa` — filtered search (date range, domain, content type, category)**Firecrawl tools:**- `firecrawl_scrape` — extract clean content from a single URL (handles JS)- `firecrawl_search` — web search with content extraction- `firecrawl_crawl` — multi-page extraction from an entire site- `firecrawl_extract` — structured LLM-powered data extraction from a URL- `firecrawl_batch_scrape` — process multiple URLs efficiently- `firecrawl_agent` — autonomous web research agent for complex tasks## Workflow1. **Discover** — use `web_search_exa` to find relevant pages semantically2. **Fetch** — use `firecrawl_scrape` on specific URLs that need JS rendering or clean extraction3. **Structure** — use `firecrawl_extract` if you need JSON output from a page4. **Bulk** — use `firecrawl_batch_scrape` or `firecrawl_crawl` for multi-page tasks## Examples- "Find top fitness researchers last 3 years" → `web_search_advanced_exa` with date filter- "Scrape this LinkedIn job posting [URL]" → `firecrawl_scrape`- "Get all blog posts from firecrawl.dev" → `firecrawl_crawl`- "Extract structured job data from this page" → `firecrawl_extract`- "Who is writing about RF antenna design right now?" → `web_search_exa`

Frequently asked questions

Web Scraping