# Web Scraping

> Scrapes webpages for text and images using a headless browser (JS-rendered) or static HTML. Downloads images and files from the web.

- Skill: `rotoslider/web-scraping` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add rotoslider/web-scraping`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rotoslider/web-scraping/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: rotoslider (https://skillmd.com/u/rotoslider)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rotoslider/web-scraping

---


# Web Scraping & Downloads

## When to Use
- Scrape page text + images (JS-rendered) -> `scrape_page_content` (best for modern sites, SPAs, product pages)
- Find images on a simple page -> `scrape_page_images` (faster, static HTML only)
- Download image -> `download_web_image` (after scraping for real URLs)
- Download file (PDF, doc, etc.) -> `download_web_file`

## scrape_page_content vs scrape_page_images
- `scrape_page_content`: Uses headless Chromium, renders JavaScript, extracts text AND images. Use for product pages, SPAs, dynamic content. Returns text + sorted images (largest first).
- `scrape_page_images`: Static HTML fetch, faster but misses JS-loaded content. Use for simple pages with standard img tags.

## Important
- NEVER guess CDN URLs -- always scrape first
- WebP images auto-converted to PNG on download
- Browser User-Agent headers used to avoid 403 blocks
- Image download: 10MB limit, File download: 50MB limit
- scrape_page_content text capped at 15K chars to avoid context bloat