# Web Scraping

> Web Scraping Skill

- Skill: `pencil20388-eng/web-scraping` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pencil20388-eng/web-scraping`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pencil20388-eng/web-scraping/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: pencil20388-eng (https://skillmd.com/u/pencil20388-eng)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pencil20388-eng/web-scraping

---

# Web Scraping Skill

Extract structured data from websites. Covers static scraping (fast, lightweight) and
dynamic scraping (for JavaScript-rendered pages).

## When to use what

| Approach | When to use | Speed | Libraries |
|---|---|---|---|
| Static (requests + BS4) | Server-rendered HTML, APIs | Fast | requests, beautifulsoup4, lxml |
| Dynamic (browser) | JavaScript-rendered, SPAs | Slower | playwright, selenium |

**Default to static scraping.** Only use browser-based scraping if the content is rendered by JavaScript.

## Installation

```bash
# Static scraping
pip install requests beautifulsoup4 lxml

# Dynamic scraping (if needed)
pip install playwright && playwright install chromium
```

## Static Scraping Patterns

### Basic page fetch + parse

```python
import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com", timeout=10)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "lxml")

title = soup.find("h1").text
links = [a["href"] for a in soup.select("a[href]")]
```

### Extract structured data

```python
products = []
for card in soup.select(".product-card"):
    products.append({
        "name": card.select_one(".name").text.strip(),
        "price": card.select_one(".price").text.strip(),
        "url": card.select_one("a")["href"],
    })
```

### Handle pagination

```python
all_items = []
page = 1

while True:
    resp = requests.get(f"https://example.com/products?page={page}", timeout=10)
    soup = BeautifulSoup(resp.text, "lxml")

    items = soup.select(".product-card")
    if not items:
        break  # No more pages

    for item in items:
        all_items.append(item.select_one(".name").text.strip())

    page += 1
    time.sleep(1)  # Be polite

print(f"Scraped {len(all_items)} items across {page - 1} pages")
```

### Set headers to avoid blocks

```python
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
}
resp = requests.get(url, headers=headers, timeout=10)
```

### Handle cookies and sessions

```python
session = requests.Session()
# Login first
session.post("https://example.com/login", data={"user": "x", "pass": "y"})
# Then scrape authenticated pages
resp = session.get("https://example.com/dashboard")
```

### Save to CSV

```python
import csv

with open("output.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "price", "url"])
    writer.writeheader()
    writer.writerows(products)
```

## Dynamic Scraping (JavaScript pages)

Use Playwright when the page content is rendered by JavaScript:

```python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://spa-example.com")
    page.wait_for_selector(".product-card")

    items = page.query_selector_all(".product-card")
    for item in items:
        name = item.text_content()
        print(name)

    browser.close()
```

## Anti-bot handling

| Technique | Counter-measure |
|---|---|
| Rate limiting | Add `time.sleep(1-3)` between requests |
| User-Agent check | Set a realistic User-Agent header |
| Cookie/session check | Use `requests.Session()` |
| JavaScript rendering | Use Playwright or Selenium |
| CAPTCHA | Consider using a CAPTCHA solving service, or scrape a different source |
| IP blocking | Rotate proxies |

## CSS Selector cheat sheet

| Selector | Meaning |
|---|---|
| `div.classname` | div with class |
| `#idname` | Element with ID |
| `div > p` | Direct child |
| `div p` | Any descendant |
| `a[href]` | Element with attribute |
| `a[href*="keyword"]` | Attribute contains |
| `.card:nth-child(2)` | Second child |
| `div.card + div.card` | Adjacent sibling |

## Important notes

- Always set a timeout on requests: `requests.get(url, timeout=10)`
- Respect `robots.txt` and rate limit your requests
- Use `lxml` parser over `html.parser` — it's faster and handles broken HTML better
- Check if the site has a public API first — it's always better than scraping
- Store raw HTML before parsing; if your parser fails, you don't need to re-fetch