Browser Automation Expert

Browser Automation Expert

Driving a browser to obtain data or complete a flow that has no API. Distinct from playwright-expert, which covers browsers as a testing tool against your own application; this skill covers browsers as a client against someone else's.

Before Automating: Is It Allowed?

Automation of a third-party site is a legal and ethical question before it is a technical one. Check, and record the answer:

• Is there an API? Use it. It is faster, more stable and unambiguously permitted.
• Terms of service — automated access is often restricted. Read them.
• robots.txt — not legally binding everywhere, but ignoring it is a deliberate act you should be able to justify.
• Personal data — scraping triggers GDPR obligations regardless of the data being public. There must be a lawful basis, and data subjects have rights over what you collected.
• Copyright and database rights — publicly readable is not freely reusable.
• Load — your throughput is someone else's cost.

If the answer to any of these is unclear, escalate rather than proceeding. This skill will not help evade access controls, defeat CAPTCHAs on a site that has refused you, or scrape behind an authentication wall you are not entitled to cross.

Core Concepts

Choose the Lightest Tool

Open the network tab before writing a scraper. Most "JavaScript-heavy" sites fetch their data from a JSON endpoint you can call directly — a hundred times cheaper and far more stable than driving a browser.

Selectors Are the Fragile Part

CSS classes generated by a build tool change on every deploy. Prefer, in order: stable id and data-* attributes, accessible roles and labels, text content, then structural position as a last resort.

Waiting Is Not Sleeping

sleep(3) is both slow and flaky. Wait for a condition — an element, a response, a network idle state — with a timeout.

Driving a Browser

from playwright.sync_api import sync_playwright

def fetch_listings(url: str) -> list[dict]:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(
            user_agent="ResearchBot/1.0 (+https://example.com/bot)",
            locale="en-GB",
            viewport={"width": 1280, "height": 900},
        )
        page = context.new_page()
        page.goto(url, wait_until="domcontentloaded", timeout=30_000)
        page.wait_for_selector("[data-testid='listing']", timeout=15_000)

        listings = [
            {
                "title": el.get_attribute("data-title"),
                "price": el.locator(".price").inner_text().strip(),
                "url": el.locator("a").get_attribute("href"),
            }
            for el in page.locator("[data-testid='listing']").all()
        ]
        context.close()
        browser.close()
        return listings

Identify your bot honestly in the user agent, with a contact URL. It is the difference between being rate-limited and being blocked, and it is the professional default.

Intercepting the data directly

Often the fastest correct approach: let the page load and capture its API response rather than parsing the DOM.

def capture_api(url: str) -> dict:
    payload = {}

    def on_response(response):
        if "/api/v2/search" in response.url and response.ok:
            payload.update(response.json())

    page.on("response", on_response)
    page.goto(url, wait_until="networkidle")
    return payload

The JSON is already structured, already typed, and does not break when the layout changes.

Blocking what you do not need

Images, fonts and analytics account for most of the bandwidth and time.

context.route("**/*", lambda route: (
    route.abort() if route.request.resource_type in {"image", "font", "media"}
    else route.continue_()
))

Authentication

Log in once, reuse the session, and never re-authenticate per request.

# One-off: perform the login and persist the session
context = browser.new_context()
page = context.new_page()
page.goto("https://example.com/login")
page.fill("#username", os.environ["SITE_USERNAME"])
page.fill("#password", os.environ["SITE_PASSWORD"])
page.click("button[type=submit]")
page.wait_for_url("**/dashboard")
context.storage_state(path="state.json")          # cookies + localStorage

# Later runs
context = browser.new_context(storage_state="state.json")

Credentials come from the environment or a secrets manager. state.json is a credential: keep it out of version control, encrypt it at rest, and set an expiry so a stale session fails loudly rather than silently returning logged-out pages.

Crawling Politely

import asyncio, urllib.robotparser
from urllib.parse import urljoin, urlparse

class Crawler:
    def __init__(self, root: str, concurrency: int = 2, delay: float = 1.0):
        self.root, self.delay = root, delay
        self.seen: set[str] = set()
        self.sem = asyncio.Semaphore(concurrency)
        self.robots = urllib.robotparser.RobotFileParser()
        self.robots.set_url(urljoin(root, "/robots.txt"))
        self.robots.read()

    def allowed(self, url: str) -> bool:
        return (
            urlparse(url).netloc == urlparse(self.root).netloc     # stay on the site
            and self.robots.can_fetch("ResearchBot", url)
        )

    async def crawl(self, url: str, depth: int = 0, max_depth: int = 3):
        if depth > max_depth or url in self.seen or not self.allowed(url):
            return
        self.seen.add(url)
        async with self.sem:
            page_data, links = await self.fetch(url)
            await asyncio.sleep(self.delay)                        # between requests
        yield page_data
        for link in links:
            async for item in self.crawl(link, depth + 1, max_depth):
                yield item

Low concurrency and a real delay are not politeness theatre — they keep you below the threshold that gets an IP range banned, and they respect the fact that a small site's server is not free.

Honour Crawl-delay and Retry-After when present. On 429 or 503, back off exponentially rather than retrying immediately.

Anti-Bot Measures

Sites signal how they want to be treated. Read the signal.

• Rate limiting (429) — slow down. This is a request, and complying is usually enough.
• A CAPTCHA appearing occasionally — reduce rate, use a persistent session, behave less mechanically.
• A hard block, an explicit refusal, or a CAPTCHA on every request — the site has declined. Stop and seek permission or a data licence.

Legitimate technical hygiene that also reduces false positives: reuse a session rather than opening a fresh browser per page, keep a consistent and honest user agent, request at human-plausible intervals with jitter, do not parallelise aggressively against one host, and cache so you never fetch the same page twice.

Circumventing an access control that has been applied to you specifically is out of scope for this skill, whatever the technique.

Extraction

from selectolax.parser import HTMLParser

def parse_listing(html: str) -> dict:
    tree = HTMLParser(html)

    def text(selector: str) -> str | None:
        node = tree.css_first(selector)
        return node.text(strip=True) if node else None

    return {
        "title": text("h1[itemprop=name]"),
        "price": parse_money(text("[data-price]")),
        "sku": text("[itemprop=sku]"),
    }

Check for structured data before parsing the DOM — many sites publish application/ld+json blocks that give you clean, typed records for free.

Validate what you extract. A scraper that silently returns None for price across ten thousand records produces a dataset that looks fine and is worthless.

class Listing(BaseModel):
    title: str = Field(min_length=1)
    price: Decimal = Field(gt=0)
    sku: str

Best Practices

• Cache aggressively. Store raw HTML with its URL and timestamp; re-parse from cache while developing selectors rather than re-fetching.
• Make it resumable. Persist progress so a crawl interrupted at 80 % does not restart.
• Monitor for silent breakage. Alert when the extraction success rate drops — layout changes fail quietly.
• Run browsers in a container with memory limits, and always close contexts; leaked browser processes exhaust a host quickly.
• Record the fetch date with every record. Scraped data ages.
• Set a per-page timeout and an overall run budget.
• Keep the parser separate from the fetcher so you can re-parse a cached corpus without touching the network.

Anti-Patterns

• Using a browser for static HTML — fifty times the cost for no benefit.
• sleep() instead of waiting for a condition — slow and flaky at once.
• Selectors on generated class names — .css-1x9y2z breaks on the next deploy.
• Unbounded concurrency — a denial of service you did not intend to commit.
• Ignoring 429 and retrying harder — escalates a rate limit into a ban.
• Storing personal data without a lawful basis — a compliance incident, not a technical one.
• No validation on extracted fields — silent corruption at scale.

Reference Documentation

• Playwright Recipes — infinite scroll, pagination, file downloads and uploads, iframes, shadow DOM, PDFs and screenshots, containerised deployment

Resources

• Playwright documentation
• Puppeteer documentation
• robots.txt specification (RFC 9309)
• EDPB guidance on web scraping and GDPR
• schema.org structured data