Scrapfly Browser

Scrapfly Cloud Browser

Use the Scrapfly Cloud Browser API with Python Playwright to automate remote cloud browsers with built-in proxy rotation, anti-bot fingerprinting, and geo-targeting.

When to use

• Automating browser interactions on remote cloud browsers (login flows, form filling, navigation)
• Scraping JavaScript-heavy sites that require full browser automation
• Bypassing anti-bot protections with managed browser fingerprinting
• Running Playwright scripts through geo-targeted proxies
• Maintaining persistent browser sessions across multiple connections
• Downloading files through browser interactions

Setup

pip install playwright

The API key must be provided via environment variable SCRAPFLY_API_KEY or passed directly in the connection URL.

API Reference

Connection: WebSocket CDP (Chrome DevTools Protocol)

wss://browser.scrapfly.io?api_key=YOUR_API_KEY

Connection Parameters

Building the Connection URL

import os
from urllib.parse import urlencode

params = {
    "api_key": os.environ["SCRAPFLY_API_KEY"],
    "proxy_pool": "datacenter",
    "os": "linux",
}
BROWSER_WS = f"wss://browser.scrapfly.io?{urlencode(params)}"

Examples

Basic page navigation

from playwright.sync_api import sync_playwright
import os

API_KEY = os.environ["SCRAPFLY_API_KEY"]
BROWSER_WS = f"wss://browser.scrapfly.io?api_key={API_KEY}&proxy_pool=datacenter&os=linux"

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        page.goto("https://web-scraping.dev")
        print("Title:", page.title())
        print("Content:", page.content()[:500])
    finally:
        browser.close()

Async Playwright

from playwright.async_api import async_playwright
import asyncio
import os

API_KEY = os.environ["SCRAPFLY_API_KEY"]
BROWSER_WS = f"wss://browser.scrapfly.io?api_key={API_KEY}&proxy_pool=datacenter&os=linux"

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(BROWSER_WS)
        try:
            context = browser.contexts[0]
            page = context.pages[0] if context.pages else await context.new_page()

            await page.goto("https://web-scraping.dev")
            print("Title:", await page.title())
        finally:
            await browser.close()

asyncio.run(main())

Login flow with form filling

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        page.goto("https://web-scraping.dev/login")
        page.fill("input[name=username]", "user123")
        page.fill("input[name=password]", "password")
        page.click("button[type=submit]")
        page.wait_for_selector("div#secret-message")

        print("Logged in:", page.url)
    finally:
        browser.close()

Take a screenshot

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        page.goto("https://web-scraping.dev")
        page.screenshot(path="screenshot.png", full_page=True)
    finally:
        browser.close()

Scrape data with selectors

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        page.goto("https://web-scraping.dev/products")
        page.wait_for_selector("div.product")

        products = page.query_selector_all("div.product")
        for product in products:
            name = product.query_selector("a").inner_text()
            price = product.query_selector(".price").inner_text()
            print(f"{name}: {price}")
    finally:
        browser.close()

Geo-targeted browsing with residential proxy

BROWSER_WS = (
    f"wss://browser.scrapfly.io?"
    f"api_key={API_KEY}"
    f"&proxy_pool=residential"
    f"&country=de"
    f"&os=windows"
)

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        page.goto("https://web-scraping.dev")
        print("Page content from DE proxy:", page.title())
    finally:
        browser.close()

Persistent sessions (multi-step workflows)

Uses the same login credentials and selectors as the Login flow with form filling example.

LOGIN_URL = "https://web-scraping.dev/login"
USERNAME = "user123"
PASSWORD = "password"

# Step 1: Login and keep session alive
BROWSER_WS = (
    f"wss://browser.scrapfly.io?"
    f"api_key={API_KEY}"
    f"&session=my-session"
    f"&auto_close=false"
    f"&proxy_pool=datacenter"
)

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        page.goto(LOGIN_URL)
        page.fill("input[name=username]", USERNAME)
        page.fill("input[name=password]", PASSWORD)
        page.click("button[type=submit]")
        page.wait_for_selector("div#secret-message")
        print("Login complete, session preserved")
    finally:
        browser.close()  # Disconnects but session stays alive

# Step 2: Reconnect to same session (still authenticated with same credentials)
BROWSER_WS = (
    f"wss://browser.scrapfly.io?"
    f"api_key={API_KEY}"
    f"&session=my-session"
    f"&proxy_pool=datacenter"
)

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        page.goto(LOGIN_URL)
        page.wait_for_selector("div#secret-message")
        print("Accessed protected page (still logged in):", page.title())
    finally:
        browser.close()

Infinite scroll scraping

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        page.goto("https://web-scraping.dev/testimonials")
        page.wait_for_selector("span.rating")

        all_items = []
        for _ in range(10):  # Scroll 10 times
            items = page.query_selector_all("span.rating")
            all_items = items
            page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            page.wait_for_timeout(2000)

        print(f"Collected {len(all_items)} reviews")
    finally:
        browser.close()

Execute JavaScript in page

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        page.goto("https://web-scraping.dev")

        # Extract data using JavaScript
        data = page.evaluate("""
            () => {
                return {
                    title: document.title,
                    links: Array.from(document.querySelectorAll('a')).map(a => ({
                        text: a.innerText,
                        href: a.href,
                    })),
                };
            }
        """)
        print(f"Found {len(data['links'])} links")
    finally:
        browser.close()

Multi-page navigation

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    try:
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()

        urls = [
            "https://web-scraping.dev/products",
            "https://web-scraping.dev/products?page=2",
            "https://web-scraping.dev/products?page=3",
        ]

        results = []
        for url in urls:
            page.goto(url)
            results.append({"url": url, "content": content})

        print(f"Scraped {len(results)} pages")
    finally:
        browser.close()

Anti-Bot Features

The Cloud Browser automatically manages browser fingerprinting:

• TLS/JA3 fingerprints matching real browsers
• User-Agent, device metrics, timezone, locale (based on country)
• WebGL and Canvas fingerprint emulation
• HTTP/2 fingerprints

BYOP (Bring Your Own Proxy) Exit Peers

For sites that allow-list the customer's own corporate IP range (vendor portals, supplier extranets, internal apps behind a VPN), the customer can run a scrapfly-byop-connector binary inside their network and route browser traffic out of that machine's egress IP.

How a Cloud Browser session opts in:

config = ScrapeConfig(
    url="https://allow-listed-portal.example.com/login",
    render_js=True,
    cloud_browser=CloudBrowserConfig(
        # Format: socks5h://proxy-<short_id>:<api_key>@proxy.scrapfly.io:1080
        byop_proxy=f"socks5h://proxy-{short_id}:{api_key}@proxy.scrapfly.io:1080",
    ),
)

Note: the SDK exposure (byop_proxy=... field on CloudBrowserConfig) is not yet implemented in any SDK. Today the field works at the API layer; SDK callers must construct the wss:// URL by hand. Customer doc: docs/rpa/exit-peers.

When to use BYOP:

• The target site allow-lists IPs and only the customer's range is on the list.
• Compliance or audit requires the customer to control their own egress.
• HIPAA / PHI workflows where the patient-portal session must stay inside the customer's TLS boundary.

How customers install the connector:

# One-liner. Auto-detects OS/arch, downloads binary + 7-day cert, unpacks under ~/.scrapfly/byop
curl -fsSL "https://scrapfly.io/install/byop-connector.sh?api_key=YOUR_API_KEY" | sh

# Or via the CLI (same end state)
scrapfly exit-peer install
scrapfly exit-peer run --name $(hostname)
scrapfly exit-peer status

After the connector is running, the dashboard at /dashboard/rpa/exit-peers shows it with a green Live badge and a proxy-<short_id> you can wire into an RPA Profile (kind: byop, connector: proxy-<short_id>). Any workflow or AI agent run that uses that profile routes browser traffic out through the customer's machine.

Important Notes

• Always use connect_over_cdp() (not connect()) to connect to the Cloud Browser
• Always use browser.contexts[0] to get the pre-configured context. Do NOT create new contexts - the Cloud Browser provides a context with proper fingerprinting
• Always close the browser in a try/finally block to stop billing. Forgetting to close means continuous charges
• Do NOT set fingerprint overrides (user agent, viewport, timezone, etc.) - they are managed automatically
• Sessions expire after 1 hour of inactivity
• Maximum session duration is 30 minutes (1800 seconds)
• Maximum file download size is 100 MB per file
• Start with proxy_pool=datacenter, upgrade to residential only when needed for anti-bot bypass
• Use debug=true for session video recording when troubleshooting