How do I install the Web Scraper skill?

Run npx skillmds@latest add dalehurley/web-scraper in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Web Scraper safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Web Scraper?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Web Scraper free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Web Scraper?

dalehurley (@dalehurley) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Web Scraper by dalehurley — Agent Skill

Name: Web Scraper
Author: dalehurley

Web Scraper

Fetch any URL and extract clean readable content as text or markdown. Use this skill when the user asks to scrape a webpage, extract text from a URL, fetch website content, read an article from a link, or download webpage content.

dalehurley a4e1f5b 2 files · 7.5 KB Updated Sep 17, 2026

File contents

Skill: web-scraper

When to Use

Use this skill when the user asks to:

Scrape or fetch content from a URL
Extract text from a webpage
Read an article from a link
Get the content of a website
Download and convert a webpage to text/markdown
Summarize a web page

Input Parameters

Parameter	Required	Description	Example
`url`	Yes	URL to scrape	https://example.com/article
`format`	No	Output format: `markdown` (default), `text`, `html`	markdown
`output`	No	Save output to file path	article.md

Procedure

Get the URL from the user's request

Run the bundled script:

# Fetch and extract as markdown (default)
python3 skills/web-scraper/scripts/scrape.py "https://example.com/article"

# Extract as plain text
python3 skills/web-scraper/scripts/scrape.py "https://example.com" --format text

# Save to file
python3 skills/web-scraper/scripts/scrape.py "https://example.com" --output article.md

The script auto-installs trafilatura and requests if needed
Present the extracted content to the user

Bundled Scripts

Script	Type	Description
`scripts/scrape.py`	Python	Fetch URL and extract clean content

Script Usage

# Extract article content as markdown
python3 scripts/scrape.py "https://example.com/blog-post"

# Extract as plain text
python3 scripts/scrape.py "https://example.com" --format text

# Keep raw HTML
python3 scripts/scrape.py "https://example.com" --format html

# Save to file
python3 scripts/scrape.py "https://example.com" --output page.md

# Include metadata (title, author, date)
python3 scripts/scrape.py "https://example.com/article" --metadata

Example

scrape this URL: https://example.com/article
extract the text from this webpage
fetch the content of this link and save it
read this article: https://blog.example.com/post-1
get the content from this website

---name: web-scraperdescription: Fetch any URL and extract clean readable content as text or markdown. Use this skill when the user asks to scrape a webpage, extract text from a URL, fetch website content, read an article from a link, or download webpage content.---# Skill: web-scraper## When to UseUse this skill when the user asks to:- Scrape or fetch content from a URL- Extract text from a webpage- Read an article from a link- Get the content of a website- Download and convert a webpage to text/markdown- Summarize a web page## Input Parameters| Parameter | Required | Description | Example || --------- | -------- | --------------------------------------------------- | --------------------------- || `url` | Yes | URL to scrape | https://example.com/article || `format` | No | Output format: `markdown` (default), `text`, `html` | markdown || `output` | No | Save output to file path | article.md |## Procedure1. Get the URL from the user's request2. Run the bundled script: ```bash # Fetch and extract as markdown (default) python3 skills/web-scraper/scripts/scrape.py "https://example.com/article" # Extract as plain text python3 skills/web-scraper/scripts/scrape.py "https://example.com" --format text # Save to file python3 skills/web-scraper/scripts/scrape.py "https://example.com" --output article.md ```3. The script auto-installs `trafilatura` and `requests` if needed4. Present the extracted content to the user## Bundled Scripts| Script | Type | Description || ------------------- | ------ | ----------------------------------- || `scripts/scrape.py` | Python | Fetch URL and extract clean content |### Script Usage```bash# Extract article content as markdownpython3 scripts/scrape.py "https://example.com/blog-post"# Extract as plain textpython3 scripts/scrape.py "https://example.com" --format text# Keep raw HTMLpython3 scripts/scrape.py "https://example.com" --format html# Save to filepython3 scripts/scrape.py "https://example.com" --output page.md# Include metadata (title, author, date)python3 scripts/scrape.py "https://example.com/article" --metadata```## Example```scrape this URL: https://example.com/articleextract the text from this webpagefetch the content of this link and save itread this article: https://blog.example.com/post-1get the content from this website```

Frequently asked questions