Browser Automation

Browser Automation

Browser automation using Playwright for scraping, screenshots, and form interactions.

When to Use

• User wants to scrape web data (e.g., Aeon article lists)
• User wants to screenshot a webpage
• User wants to automate form filling
• User wants to log into a website and perform actions
• Standard fetch tools cannot access dynamically loaded content

Install Dependencies

On first use, install Playwright:

# Python approach
pip install playwright
playwright install chromium

# Or Node approach
npm install playwright
npx playwright install chromium

Core Capabilities

1. Page Screenshot

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page(viewport={"width": 1280, "height": 800})
    page.goto("https://example.com")
    page.screenshot(path="screenshot.png", full_page=True)
    browser.close()

2. Data Scraping

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://aeon.co/essays")

    # Wait for content to load
    page.wait_for_selector(".essay-card")

    # Extract data
    essays = page.query_selector_all(".essay-card")
    for essay in essays:
        title = essay.query_selector("h2").inner_text()
        link = essay.query_selector("a").get_attribute("href")
        print(f"{title}: {link}")

    browser.close()

3. Form Filling & Login

page.goto("https://example.com/login")
page.fill("#username", "user@example.com")
page.fill("#password", "password123")
page.click("button[type=submit]")
page.wait_for_load_state("networkidle")

4. PDF Generation

page.goto("https://example.com/article")
page.pdf(path="article.pdf", format="A4")

Important Rules

1. Try fetch tools first — For static pages, prefer built-in fetch/web tools
2. Only launch browser when necessary — Dynamic content, interaction needed, login required
3. Set reasonable timeouts — Default 30 seconds, extend to 60 seconds for complex pages
4. Use headless mode — Do not display browser window by default
5. Close immediately after scraping — Avoid resource leaks

Common Use Cases

Scraping Aeon Article List

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://aeon.co/essays")
    page.wait_for_selector("article")

    articles = page.query_selector_all("article")
    results = []
    for article in articles:
        title_el = article.query_selector("h2 a, h3 a")
        if title_el:
            results.append({
                "title": title_el.inner_text(),
                "url": title_el.get_attribute("href")
            })

    browser.close()

Scrolling for Infinite-Load Pages

# Scroll to load more content
for i in range(5):  # Scroll 5 times
    page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    page.wait_for_timeout(2000)  # Wait 2 seconds

Troubleshooting