Web Scraping Javascript Sites

Web Scraping JavaScript/SPA Sites

The Core Insight

For SPA/TypeScript sites: try urllib FIRST before launching a browser.

TypeScript sites (Next.js, Nuxt, React, Vue) send full HTML on first request — the content is there. Browser tools show empty snapshots because they capture before JS finishes rendering. A raw HTTP fetch often gets all the data you need.

Decision Tree

Is the site an SPA (React/Next/Nuxt/Vue)?
│
├── YES → Try urllib/requests FIRST
│         └─ Got content? → DONE
│         └─ Content empty? → Use Playwright
│
└── NO (static HTML) → Use urllib/requests directly

Layer 1: Raw HTML Fetch (Fastest)

import urllib.request
import re

url = "https://example.com"
req = urllib.request.Request(url, headers={
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
})
response = urllib.request.urlopen(req, timeout=10)
html = response.read().decode('utf-8')

# Extract content
# Option A: regex search
prices = re.findall(r'€[\d,]+', html)

# Option B: find section context
section = re.search(r'id="pricing".{0,5000}', html, re.DOTALL | re.IGNORECASE)
if section:
    content = re.sub(r'<[^>]+>', ' ', section.group())
    content = re.sub(r'\s+', ' ', content).strip()

# Option C: parse HTML properly
from html.parser import HTMLParser
class ContentParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.in_target = False
        self.text = []
    def handle_starttag(self, tag, attrs):
        if tag in ('h1','h2','h3','p', 'li'):
            self.in_target = True
    def handle_data(self, data):
        if self.in_target:
            self.text.append(data.strip())
    def handle_endtag(self, tag):
        if tag in ('h1','h2','h3','p', 'li'):
            self.in_target = False

Why urllib Often Works for "Lazy" Sites

Many TypeScript/React sites pre-render HTML on the server (SSR) or send full content in initial HTML, but use JS to make it look animated/lazy. The data is in the first response — you just need to parse it.

Layer 2: Playwright (When Needed)

Use when:

• Content loads via AJAX/fetch calls AFTER page load
• Content requires user interaction (click to expand, scroll, login)
• Site has strong bot detection
• Single-page app with client-side routing

Setup

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    
    # Human-like headers
    page.set_extra_http_headers({
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
    })
    page.set_viewport_size({"width": 1920, "height": 1080})
    
    # Navigate and wait properly
    page.goto(url, wait_until='networkidle', timeout=30000)
    
    # Wait for specific content (NOT just page load)
    page.wait_for_selector(".pricing-card", timeout=10000)
    
    # Extract
    content = page.text_content("body")
    
    browser.close()

Waiting Patterns (Humanize)

# ❌ Robot: instant read after click
page.click("button")
result = page.text_content(".result")  # Empty!

# ✅ Human: wait for DOM change
page.click("button")
page.wait_for_selector(".result:not(:empty)", timeout=5000)
result = page.text_content(".result")

# ❌ Robot: no wait
page.goto(url)
title = page.text_content("h1")

# ✅ Human: wait for content
page.goto(url)
page.wait_for_selector("h1", timeout=10000)
title = page.text_content("h1")

Scroll Like a Human

for _ in range(3):
    page.mouse.wheel(0, 500)
    page.wait_for_timeout(300)

Hover Before Click (Pass Sophisticated Bot Detection)

page.hover("nav .menu-item")
page.wait_for_timeout(100)
page.click("nav .menu-item")

Stealth Mode (Puppeteer Extra)

For sites with bot detection:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

(async () => {
  const browser = await puppeteer.launch({ 
    headless: 'new',
    args: ['--no-sandbox', '--disable-setuid-sandbox']
  });
  const page = await browser.newPage();
  await page.goto('https://bot-detected-site.com');
  // ...
})();

Available at: $HERMES_AGENT_DIR/node_modules/puppeteer-extra-plugin-stealth

TypeScript Site Patterns

Next.js Static Export

# These are fully scrapeable with urllib - HTML is complete
page.goto("https://site.com/page")
# Content is in initial HTML

Next.js SSR

# Server renders full HTML - urllib works
page.goto("https://site.com/page")

Client-Side Only (CRA/Vite)

# Must use Playwright - no HTML content without JS
page.goto("https://site.com/page")
page.wait_for_selector(".content")

Nuxt/Vue

# Often SSR with hydration - urllib works
page.goto("https://site.com/page")

Common Issues

Bot Detection Bypass

# Try these headers
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-User': '?1',
    'Cache-Control': 'max-age=0',
}

Tools Available on Appie-3 VPS

Quick Test Script

#!/usr/bin/env python3
"""Test if a site is scrapeable with urllib or needs Playwright"""

import urllib.request
import sys

def test_site(url):
    req = urllib.request.Request(url, headers={
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
    })
    try:
        response = urllib.request.urlopen(req, timeout=10)
        html = response.read().decode('utf-8')
        print(f"SUCCESS: {len(html)} bytes received")
        
        # Check for meaningful content
        if len(html) < 5000:
            print("WARNING: Very small HTML - likely needs JS rendering")
            return False
        
        # Check for text content
        import re
        text = re.sub(r'<[^>]+>', '', html)
        text = re.sub(r'\s+', ' ', text).strip()
        print(f"Text content: {len(text)} chars")
        
        if text.count(' ') < 50:
            print("WARNING: Little text content - likely JS-rendered SPA")
            return False
            
        return True
    except Exception as e:
        print(f"ERROR: {e}")
        return False

if __name__ == "__main__":
    url = sys.argv[1] if len(sys.argv) > 1 else "https://example.com/app"
    print(f"Testing: {url}")
    can_scrape = test_site(url)
    print(f"\nRecommendation: {'urllib works' if can_scrape else 'Use Playwright'}")

Run: python3 $HERMES_SKILLS_DIR/web-scraping-javascript-sites/scripts/test-site.py https://example.com

See Also

• skills/playwright — Full Playwright reference
• skills/ui-ux-pro-max — For taking screenshots of scraped sites