AI agent skills for web scraping and research
Scraping and research skills give the agent a method for gathering: what to collect, how to cite it, and when a source is not good enough to use. The research-oriented ones are the more valuable half — an agent that records where each claim came from produces work you can check. Anything that fetches pages declares network access on its page, which is worth reading before you install it.
People land here searching for “claude code web scraping skill”, “agent skill for research”, “skill to summarize a website”.
9 web scraping and research skills Refine in search →
-
carlvellotti Skill Web ResearchWeb research using Tavily and Firecrawl. Searches the web, reads pages, crawls sites, and synthesizes findings. Use when: (1) user says "web research", "research this", "look into", "find out about", "what's the latest on", "/web-research", (2) user needs information from the web to inform a decision, strategy, or content, (3) user provides a URL and wants it read/analyzed, (4) user wants competitive intel, market research, or topic deep-dives. NOT for: X/Twitter research, Reddit research, scraping for data pipelines, or browser automation unrelated to research.
- carlvellotti Skill Web Research 2
Web research using Tavily and Firecrawl. Searches the web, reads pages, crawls sites, and synthesizes findings. Use when: (1) user says "web research", "research this", "look into", "find out about", "what's the latest on", "/web-research", (2) user needs information from the web to inform a decision, strategy, or content, (3) user provides a URL and wants it read/analyzed, (4) user wants competitive intel, market research, or topic deep-dives. NOT for: X/Twitter research, Reddit research, scraping for data pipelines, or browser automation unrelated to research.
-
nousresearch Skill ScraplingScrape sites with stealth browsing and Cloudflare bypass.
-
dabit3 Skill ScraplingWeb scraping with Scrapling - HTTP fetching, stealth browser automation, Cloudflare bypass, and spider crawling via CLI and Python.
Audited -
angelburgosrosado Skill ScraplingWeb scraping with Scrapling - HTTP fetching, stealth browser automation, Cloudflare bypass, and spider crawling via CLI and Python.
- angelburgosrosado Skill Scrapling 2
Web scraping with Scrapling - HTTP fetching, stealth browser automation, Cloudflare bypass, and spider crawling via CLI and Python.
-
braxtonrose4 Skill ScraplingWeb scraping with Scrapling - HTTP fetching, stealth browser automation, Cloudflare bypass, and spider crawling via CLI and Python.
Audited -
dvcrn Bundle Time MasheenTHE_TIME_MASHEEN — full-spectrum web intelligence combining live scraping, historical time travel, and interactive browser automation. Use when: (1) scraping or crawling any live website (static, JS-heavy, or rendering-protected), (2) going back in time to retrieve archived or historical versions of any page via the Wayback Machine, (3) comparing what a site looks like now vs. what it looked like in any previous year, (4) automating browser interactions (login, click, fill forms) on web apps that can't be passively scraped, (5) extracting data from login-gated or paywalled pages, or (6) any task requiring live scraping + historical research + browser interaction in combination. Triggers on: "scrape this", "crawl", "wayback", "archive", "what did this site look like", "compare current vs historical", "browser automation", "extract data from", "log in and scrape", "go back in time on".
32 -
crawlora-org Bundle Kohls ResearchResearches Kohl's catalog by browsing its category taxonomy (products, prices, ratings, and facets), pulls product reviews by web_id, finds nearby Kohl's stores, and returns search-box typeahead suggestions — all via the Crawlora API as clean JSON. Use when the user asks to browse a Kohl's category, discover Kohl's product ideas for a query, pull reviews for a specific Kohl's item, or find nearby Kohl's stores — instead of scraping Kohls.com.
How to install a web scraping and research skill
- Compare the skills. Read the web scraping and research skills below — each page shows the full SKILL.md, its safety verdict, and the capabilities it declares.
- Install it. Run npx skillmds@latest add <owner>/<name>. The CLI writes the skill into every agent directory it detects, or use --agent to pin one.
- Use it. Restart your agent. It loads the skill on demand the next time you ask for something that matches — you do not have to name the skill.
What makes a good web scraping and research skill?
Scraping and research skills give the agent a method for gathering: what to collect, how to cite it, and when a source is not good enough to use. The research-oriented ones are the more valuable half — an agent that records where each claim came from produces work you can check. Anything that fetches pages declares network access on its page, which is worth reading before you install it.
Every skill listed here is a plain SKILL.md file in the format Anthropic documents for Agent Skills, read unchanged by Cursor, OpenAI Codex and 60+ agents. Each passes a safety review before it is publicly listed; the verdict and declared capabilities are on every skill's page.
Frequently asked questions
What is an agent skill for web scraping and research?
Scraping and research skills give the agent a method for gathering: what to collect, how to cite it, and when a source is not good enough to use. The research-oriented ones are the more valuable half — an agent that records where each claim came from produces work you can check. Anything that fetches pages declares network access on its page, which is worth reading before you install it.
Is scraping with an agent skill allowed?
That depends on the site, not the skill — robots.txt and terms of service still apply, and a skill cannot grant permission the site has not given. Well-written scraping skills check robots.txt and rate-limit themselves; that behaviour should be visible in the SKILL.md before you install it.
Which agents can use these web scraping and research skills?
Any agent that reads SKILL.md files: Claude Code, Claude.ai, Cursor, OpenAI Codex, Windsurf, OpenCode and 60+ others. The format is not vendor-specific, so the same file works everywhere — each agent just keeps its skills in a different directory, listed at /agents.
Are these web scraping and research skills free?
Yes. Searching, reading and installing skills on SkillMD is free and needs no account. Individual skills carry their own licence, shown on each skill's page.