Web Scraper

Scrape and extract content from web pages. Supports extracting text, links, headings, and structured data. Use when the user asks to read a web page, extract information from a URL, or scrape website content.

ericwang915 16c2eef 2 files · 3.4 KB Updated

File contents

Instructions

Scrape and extract readable content from any web page.

Prerequisites

Install dependencies: pip install requests beautifulsoup4

Usage

python {skill_path}/scrape.py URL [--format text|json|links|headings]

Formats:

  • text (default) — cleaned readable text
  • json — structured JSON with title, text, links, headings
  • links — all links on the page
  • headings — all headings (h1–h6)

Examples

Resources

File Description
scrape.py Generic web page scraper

ericwang915/PythonClaw/tree/main/pythonclaw/templates/skills/data/scraper commit 16c2eef4ff

Frequently asked questions

npx skillmds@latest add ericwang915/web-scraper