Web Scraper SEO

Web Scraper & SEO Analyzer

Scrapes websites, downloads their assets (HTML, CSS, JS, images, fonts), rewrites all paths for local serving (including internal page links and inline styles), extracts SEO metadata and design system patterns, detects broken links, records redirect chains, performs keyword density analysis, and produces a fully navigable local mirror. Scraped content can be served via a built-in HTTP server and verified with Playwright screenshots. Design tokens can be extracted in CSS custom properties, W3C JSON, or Tailwind config format.

When This Skill Fires

• User provides a URL (or list of URLs) and wants to capture, clone, or analyze the site
• User wants design inspiration or design tokens from a live website
• User asks to reverse-engineer a page's structure or design system
• User wants to audit SEO techniques on one or more pages
• User wants a local copy of a site for offline reference or mockup generation
• User wants to scrape a JavaScript-rendered SPA (React, Next.js, Vue, Angular)
• User wants to detect broken links or analyze redirect chains
• User wants to extract design tokens (colors, spacing, typography) from a site
• User wants an accessibility audit or WCAG compliance check on a website

Parameters

The user provides:

1. url (required): A single URL string or a list of URLs to scrape
2. depth (optional, default: 0): How many levels of child/navigation links to follow
   • 0 = single page only
   • 1 = the page + all direct child links found in navigation menus
   • 2 = children + grandchildren
   • 3+ = deeper crawls (warn the user this can be large)
3. render-js (optional, default: off): Use Playwright to render JS before scraping — captures SPA/React/Vue content invisible to HTTP-only scraping. Also enables shadow DOM flattening and same-domain iframe extraction
4. sitemap (optional, default: off): Fetch sitemap.xml to discover pages not linked from navigation
5. resume (optional, default: off): Resume an interrupted crawl from the last checkpoint
6. proxy (optional): Route all requests through an HTTP/HTTPS/SOCKS5 proxy (e.g., socks5://proxy:1080)
7. cookies (optional): Inject auth cookies — inline key=value;key=value pairs or a path to a JSON file {"name": "value"}
8. headers (optional): Inject custom HTTP headers — Key:Value;Key:Value format (e.g., Authorization:Bearer token123)
9. basic-auth (optional): HTTP Basic Auth as user:password

Which Script to Use

Error Handling

The scraper handles common issues gracefully — know these before running:

• 403/401 responses: Logged, page skipped, crawl continues
• Timeout: 30-second timeout per request, retries once, then skips
• SSL errors: Falls back to verify=False, warns user
• Binary files: Detected and downloaded without HTML parsing
• Relative URLs: All resolved against the page's base URL
• JavaScript-rendered content: Use --render-js to render pages with Playwright before processing. Falls back to HTTP-only if Playwright is not installed.
• Interrupted crawls: Use --resume to continue from where you left off. Checkpoint saved after every page.
• Broken links: Automatically detected and reported in broken-links.json with source pages and anchor text
• Redirect chains: Automatically recorded in redirect-chains.json with hop count and status codes
• Cookie/consent overlays: In --render-js mode, common consent banners (OneTrust, CookieBot, Osano, Quantcast, generic GDPR) are automatically dismissed before content extraction. Also applies to Playwright screenshots.
• Proxy routing: Use --proxy to route through HTTP/HTTPS/SOCKS5 proxies. Applied to both HTTP requests and Playwright browser.
• Authentication: Use --cookies, --headers, or --basic-auth to access login-protected pages. Tokens/cookies must be obtained externally.
• iframe content: Same-domain <iframe> sources are fetched and saved to pages/_iframes/. Cross-origin iframes are skipped.
• Shadow DOM: In --render-js mode, shadow DOM content is flattened into the page HTML before extraction (best-effort).

Workflow Steps

Post-Scrape Checklist (Do This After Every Scrape)

After running scrape_site.py (Step 2), complete ALL of these before moving on — these follow-up offers are a core part of the skill's value, not optional extras:

1. Present the results — Summarize pages scraped, assets downloaded, SEO highlights, broken links found, and redirect chains recorded
2. Offer design token extraction — Ask: "I can also extract the design system (colors, fonts, spacing) into CSS custom properties, W3C JSON, or Tailwind config. Want me to run that?" Then run extract_tokens.py if they say yes (or if they originally asked about design/styling)
3. Offer accessibility audit — Ask: "I can run a WCAG 2.1 accessibility audit on the scraped pages. Want me to check that?"
4. Offer local server — Ask: "Would you like me to start a local HTTP server so you can browse the scraped site in your browser?"

Even if the user didn't explicitly ask for tokens, audit, or server — offer them. Users often don't know these capabilities exist until prompted. If the user's original request already implies one of these (e.g., "clone the look and feel" implies design tokens), run it proactively without asking.

Step 1: Install Dependencies

cd /path/to/this/skill
pip install requests beautifulsoup4 lxml cssutils --break-system-packages -q
pip install playwright --break-system-packages -q
playwright install chromium 2>/dev/null || true

If playwright install fails (common in sandboxed environments), that's fine — scraping still works without JS rendering. The Playwright viewing step and --render-js become unavailable.

Step 2: Run the Scraper

python scripts/scrape_site.py \
  --urls "https://example.com" \
  --depth 1 \
  --output /home/claude/scraped-sites/example.com

For JS-heavy SPAs (React, Next.js, Vue, Angular):

python scripts/scrape_site.py \
  --urls "https://spa-site.com" \
  --depth 1 --render-js \
  --output /home/claude/scraped-sites/spa-site

With sitemap discovery for content-heavy sites (blogs, docs, e-commerce):

python scripts/scrape_site.py \
  --urls "https://docs-site.com" \
  --depth 0 --sitemap \
  --output /home/claude/scraped-sites/docs-site

Resume an interrupted crawl:

python scripts/scrape_site.py \
  --urls "https://example.com" \
  --depth 2 --resume \
  --output /home/claude/scraped-sites/example.com

Through a proxy:

python scripts/scrape_site.py \
  --urls "https://example.com" \
  --depth 1 --proxy "socks5://proxy:1080" \
  --output /home/claude/scraped-sites/example.com

Authenticated site (cookies + headers):

python scripts/scrape_site.py \
  --urls "https://dashboard.example.com" \
  --depth 1 \
  --cookies "session_id=abc123;csrf_token=xyz" \
  --headers "Authorization:Bearer eyJhbGci..." \
  --output /home/claude/scraped-sites/dashboard

The script handles:

• Breadth-first crawling with a visited-set to avoid cycles
• Depth limiting via the --depth flag
• Navigation-scoped link discovery — only follows links inside <nav>, <header>, or elements with menu/nav roles
• Sitemap discovery (--sitemap) — fetches sitemap.xml/sitemap_index.xml to find pages not in navigation
• JS rendering (--render-js) — uses Playwright to get the fully rendered DOM for SPAs before processing with the standard pipeline
• Asset downloading — CSS, JS, images, fonts from HTML and CSS @import/url() references
• Internal link rewriting — after all pages are scraped, a second pass rewrites <a href> links between pages to relative local paths, making multi-page sites fully navigable when served locally
• Inline style processing — url() references in <style> tags and style="" attributes are rewritten to local asset paths; design patterns (colors, fonts, layout) from inline styles are included in the analysis
• Keyword density analysis — top 20 keywords with density percentages, stop word filtering, and prominence detection (keywords appearing in title/H1)
• Redirect chain recording — captures the full redirect path (each hop's URL and status code) whenever HTTP redirects are followed
• Checkpoint/resume (--resume) — saves state after every page so interrupted crawls can continue from the last checkpoint
• Broken link detection — automatically tracks which pages link to which URLs and generates broken-links.json with source pages, anchor text, and error details
• Proxy routing (--proxy) — routes all HTTP requests and Playwright browser through an HTTP/HTTPS/SOCKS5 proxy
• Authentication (--cookies, --headers, --basic-auth) — injects session cookies, custom headers, or HTTP Basic Auth into both requests and Playwright contexts
• iframe extraction — same-domain <iframe> content is fetched, saved to pages/_iframes/, and the parent page's src rewritten to point locally
• Shadow DOM flattening (--render-js only) — serializes shadow root content into <template shadowrootmode> elements so downstream parsers can see web component content
• robots.txt respect, rate limiting (1 req/sec default), and honest User-Agent identification

Step 3: Understand the Output Structure

scraped-sites/example.com/
├── site-report.json          # Master manifest + SEO + design analysis
├── broken-links.json         # Broken link report with source pages + anchor text
├── redirect-chains.json      # Redirect chain data (hops, status codes)
├── pages/
│   ├── index.html            # Rewritten HTML (local asset + page link paths)
│   ├── about.html
│   └── products/
│       └── index.html
├── assets/
│   ├── css/                  # Downloaded stylesheets (url() paths rewritten)
│   ├── js/                   # Downloaded scripts
│   ├── images/               # Downloaded images, favicons, icons
│   └── fonts/                # Downloaded web fonts
├── seo/
│   ├── seo-summary.json      # Aggregated SEO findings
│   └── per-page/
│       ├── index.json        # Per-page SEO data (including keyword density)
│       └── about.json
├── a11y/                     # Generated by a11y_analyzer.py (Step 5b)
│   ├── a11y-summary.json     # Aggregated accessibility findings
│   ├── a11y-analysis.json    # Full analysis with per-page scores
│   └── per-page/
│       ├── index.json        # Per-page WCAG checks
│       └── about.json
└── design-tokens/            # Generated by extract_tokens.py (Step 5a)
    ├── tokens-raw.json       # Raw extracted values
    ├── tokens-w3c.json       # W3C Design Tokens format
    ├── tokens.css            # CSS custom properties
    └── tailwind.config.js    # Tailwind theme config

Step 4: Review the Site Report

site-report.json is the master document. It contains: scrape metadata (including JS rendering mode, sitemap discovery, broken link count, redirect chain count), a hierarchical sitemap tree, asset inventory by type, SEO highlights and issues, and design system extraction (colors, fonts, layout approach, responsive detection, framework detection).

See references/site-report-schema.md for the full schema with field descriptions and example JSON.

Step 5: Review SEO Data

seo/seo-summary.json aggregates findings across all pages. Per-page files in seo/per-page/ contain granular data covering: title tags, meta descriptions, heading hierarchy, URL structure, canonical URLs, Open Graph/Twitter Card tags, hreflang, robots directives, JSON-LD structured data, internal/external link analysis, image alt text audit, keyword density (top 20 keywords with density percentages and prominence in title/H1), and technical signals (viewport, charset, script loading, resource hints).

For the full element-by-element reference, see references/seo-checklist.md.

For deeper analysis with per-page scoring (0-100, graded A-F), run:

python scripts/seo_analyzer.py --input /path/to/scraped-site --format json

Step 5a: Extract Design Tokens (Per Post-Scrape Checklist)

Run proactively if the user asked about design, styling, or cloning — otherwise offer per the checklist above:

# All formats (CSS + JSON + Tailwind)
python scripts/extract_tokens.py --input /path/to/scraped-site --format all

# CSS custom properties only
python scripts/extract_tokens.py --input /path/to/scraped-site --format css

# W3C JSON design tokens only
python scripts/extract_tokens.py --input /path/to/scraped-site --format json

# Tailwind config only
python scripts/extract_tokens.py --input /path/to/scraped-site --format tailwind

The extractor analyzes all CSS sources (external stylesheets, inline <style> blocks, and style="" attributes on elements) and produces:

• Color classification — primary, neutral, and accent colors grouped by saturation/usage
• Typography scale — font families, font sizes sorted into a scale
• Spacing scale detection — identifies base-4, base-8, or custom spacing systems
• Border radius scale — extracted and sorted
• Shadow tokens — box-shadow values cataloged
• Breakpoint detection — media query breakpoints extracted
• CSS custom properties — ready-to-use :root { --color-primary: ... } file
• W3C Design Tokens — structured JSON following the draft W3C spec
• Tailwind config — theme.extend snippet for tailwind.config.js

Step 5b: Run Accessibility Audit (Per Post-Scrape Checklist)

Run proactively if the user asked about accessibility — otherwise offer per the checklist above:

python scripts/a11y_analyzer.py --input /path/to/scraped-site --format json

The analyzer checks each page against 10 WCAG criteria and produces a 0-100 score (graded A-F):

• Language (3.1.1) — <html lang="..."> present and valid
• Document title (2.4.2) — descriptive <title> tag
• Heading hierarchy (1.3.1) — sequential levels, single H1
• Image alt text (1.1.1) — missing alt, empty alt in links, suspicious placeholder alt
• Form labels (1.3.1, 4.1.2) — explicit <label>, aria-label, or wrapping <label>
• Link purpose (2.4.4) — no empty links, no generic "click here" text
• ARIA validity (4.1.2) — valid roles, required attributes, no hidden focusables
• Focus indicators (2.4.7) — detects outline: none without replacement
• Skip navigation (2.4.1) — skip-to-content link present
• Landmark regions (1.3.6) — <main>, <nav>, <header> presence
• Table headers (1.3.1) — <th>, scope, and <caption> on data tables
• Color contrast (1.4.3) — heuristic detection of low-contrast inline styles

Output is saved to a11y/a11y-analysis.json and a11y/a11y-summary.json, with per-page detail in a11y/per-page/.

Step 6: Offer Local Server (Per Post-Scrape Checklist)

Per the checklist above, offer the local server. If the user accepts:

python scripts/serve_site.py \
  --directory /home/claude/scraped-sites/example.com \
  --port 8080

The server maps / to pages/index.html, resolves page paths with .html extension fallback, serves /assets/* directly, and adds CORS headers.

Step 7: View with Playwright (Optional)

Use when: You need visual verification that the scrape is faithful, responsive screenshots for mockup reference, or the user wants to see the site before working with it. Skip when: The user only needs raw HTML/CSS/JSON data, or Playwright isn't installed.

# Single screenshot
python scripts/playwright_view.py screenshot \
  --url "http://localhost:8080" \
  --output screenshot.png --viewport 1280x800

# Responsive screenshots (desktop + tablet + mobile)
python scripts/playwright_view.py responsive \
  --url "http://localhost:8080" --output-dir /path/to/screenshots

Step 8: Deep Analysis with web-analyzer (Optional)

For multi-site comparison, detailed design briefs, or when the scraped CSS data is too verbose for the main conversation, delegate to the web-analyzer companion subagent. It reads the scraped output directory and returns a structured summary — keeping the main conversation context lean.

The web-analyzer handles:

• Design system deep-dive — component patterns, motion/animation, layout grid analysis
• Site comparison — side-by-side analysis of 2+ scraped sites
• Design brief generation — actionable specifications for recreating or remixing the design
• SEO recommendations — prioritized improvements with concrete examples

Post-Scrape: Using Results

With the scraped data in hand, Claude can:

1. Recreate the page — Read site-report.json for structure, reference downloaded HTML/CSS for implementation details
2. Remix the design — Use design-tokens/tokens.css or tailwind.config.js as a foundation for new projects
3. Apply SEO best practices — Use seo-summary.json as a checklist for new pages, fix issues found in broken-links.json 3a. Fix accessibility issues — Use a11y/a11y-analysis.json to identify and prioritize WCAG violations
4. Brainstorm variations — Compare multiple scraped sites side-by-side to identify patterns and differentiation opportunities
5. Fix redirect chains — Review redirect-chains.json to identify SEO-impacting redirect hops
6. Target keywords — Use keyword density data to understand content strategy and replicate it

Prioritize the design system (colors, fonts, spacing) over pixel-perfect reproduction.

Important Notes

• This skill is for learning, brainstorming, and development reference. Always respect website terms of service and copyright.
• The scraper identifies itself honestly via User-Agent and respects robots.txt.
• Large sites with deep crawls can produce substantial downloads. The skill warns when depth > 2.
• Rate limiting (1 req/sec default) prevents overwhelming target servers.
• Downloaded content is for local development reference only.

Reference Files

• scripts/scrape_site.py — Main scraping engine (crawl, download, rewrite links, JS rendering, sitemap, broken links, redirects, keyword density, cookie dismissal)
• scripts/extract_tokens.py — Design token extractor (CSS custom properties, W3C JSON, Tailwind config)
• scripts/a11y_analyzer.py — Accessibility audit (WCAG 2.1 AA, 12 checks, 0-100 scoring)
• scripts/serve_site.py — Local HTTP server for viewing scraped content
• scripts/playwright_view.py — Playwright integration for screenshots, responsive testing, JS rendering, cookie dismissal
• scripts/seo_analyzer.py — Standalone deep SEO analysis with scoring
• agents/web-analyzer.md — Companion subagent for deep design analysis and site comparison
• references/seo-checklist.md — Comprehensive SEO elements reference
• references/site-report-schema.md — Full site-report.json schema with field descriptions