Web Scraping

Web Scraping

Overview

Two ways to scrape in the sandbox:

1. MCP tools (scrape_page, scrape_pages) — recommended for straight "give me this page's content". Synchronous, return dicts.
2. Direct Scrapling Python API — for CSS/XPath selectors, sessions, logins, and multi-page spiders. Async, returns Page objects with .css() / .xpath().

Quick fetches can run inline via ExecuteCode. For spiders, multi-URL crawls, or anything you'll iterate on, write the scraper to work/<task_name>/scraper.py and run it via Bash — edit-and-rerun beats resubmitting code.

MCP Tools

Import from tools.scrape. Synchronous — no await.

from tools.scrape import scrape_page, scrape_pages

Signatures

scrape_page(url: str, mode: str = "fast", extraction: str = "markdown",
            timeout: float = 30.0, solve_cloudflare: bool = False) -> dict

scrape_pages(urls: list[str], mode: str = "fast", extraction: str = "markdown",
             timeout: float = 30.0, solve_cloudflare: bool = False) -> dict

Parameters

Escalate modes only as needed: start fast, go to browser when the page needs JavaScript, stealth when you're getting blocked, and add solve_cloudflare=True only if stealth still returns a challenge page.

Return shape

scrape_page returns a flat dict:

{
    "url": "https://example.com",
    "status": 200,
    "title": "Example Domain",
    "content": "# Example Domain\n\nThis domain is for use in...",  # str
    "extraction": "markdown",
    "mode": "fast",
}

• content is a plain string, not a list — use it directly, never content[0] (that yields a single character).
• content is truncated to 400,000 chars.
• No .css() / .xpath() / .body / .headers / .cookies — for selectors use the direct Python API below, or parse extraction="html" with BeautifulSoup.

scrape_pages wraps them:

{
    "results": [ ... ],  # one entry per input URL, in input order
    "count": 3,
}

Errors

Errors are returned, never raised. Always check for "error" before reading content.

res = scrape_page(url="https://example.com")
if "error" in res:
    print(res["error"], res["detail"])
else:
    print(res["content"])

Per-URL errors — appear as {"error", "detail", "url"} entries inside scrape_pages["results"], or as the whole return of scrape_page:

Whole-call errors — the entire return is {"error", "detail"}, no results:

One bad URL never sinks a batch. scrape_pages always returns one entry per input URL, in input order — failures come back as error entries alongside the successes.

Examples

from tools.scrape import scrape_page, scrape_pages

# Single page → markdown
res = scrape_page(url="https://example.com")
if "error" not in res:
    print(res["title"], res["status"], len(res["content"]))

# JS-rendered page
res = scrape_page(url="https://spa-site.com", mode="browser", timeout=60)

# Bot-protected page
res = scrape_page(url="https://protected-site.com", mode="stealth", solve_cloudflare=True)

# Batch — split successes from failures
batch = scrape_pages(urls=[...], mode="fast")   # <= 10 URLs
pages = [r for r in batch["results"] if "error" not in r]
failed = [(r["url"], r["error"]) for r in batch["results"] if "error" in r]

# Raw HTML when you need to parse structure yourself
res = scrape_page(url="https://example.com", extraction="html")
from bs4 import BeautifulSoup
soup = BeautifulSoup(res["content"], "html.parser")
titles = [h1.get_text() for h1 in soup.find_all("h1")]

Batches run concurrently — 8 at a time in fast mode, 2 at a time in browser / stealth (browser sessions are memory-heavy). More than 10 URLs means more than one call.

Direct Python API (Advanced)

For selectors, sessions, spiders, or when you need the full Page object. Requires imports. Async.

Fetcher (Fast HTTP — Tier 1)

from scrapling.fetchers import AsyncFetcher

page = await AsyncFetcher.get("https://example.com", stealthy_headers=True)
print(page.status)       # 200
print(page.body)         # Raw bytes
print(page.headers)      # Response headers

# CSS selectors (Scrapy-style pseudo-elements)
titles = page.css("h1::text").getall()
links = page.css("a::attr(href)").getall()

# XPath
items = page.xpath("//div[@class='item']/text()").getall()

# BeautifulSoup-style
divs = page.find_all("div", class_="content")

DynamicFetcher (Browser — Tier 2)

from scrapling.fetchers import DynamicFetcher

page = await DynamicFetcher.async_fetch(
    "https://spa-website.com",
    headless=True,
    network_idle=True,
    disable_resources=True,
    timeout=30000,          # milliseconds here, unlike the MCP tools
    wait_selector=".data-table",
)
rows = page.css("table.data-table tr")
for row in rows:
    cells = row.css("td::text").getall()

StealthyFetcher (Anti-Bot — Tier 3)

from scrapling.fetchers import StealthyFetcher

page = await StealthyFetcher.async_fetch(
    "https://protected-site.com",
    headless=True,
    solve_cloudflare=True,
    network_idle=True,
)

Sessions (Persistent Connections)

from scrapling.fetchers import FetcherSession

with FetcherSession(impersonate="chrome") as session:
    login_page = session.post("https://site.com/login", data={...})
    dashboard = session.get("https://site.com/dashboard")
    data = dashboard.css(".user-data::text").getall()

Spider (Multi-Page Crawl)

from scrapling.spiders import Spider, Request, Response

class PriceScraper(Spider):
    name = "prices"
    start_urls = ["https://example.com/products"]
    concurrent_requests = 5

    async def parse(self, response: Response):
        for product in response.css(".product"):
            yield {
                "name": product.css(".name::text").get(),
                "price": product.css(".price::text").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield Request(next_page)

spider = PriceScraper()
result = spider.start()
result.items.to_json("work/<task_name>/data/prices.json")

Converting HTML to Markdown

Only needed when you fetched HTML yourself — scrape_page(extraction="markdown") already does this.

import html_to_markdown

markdown = html_to_markdown.convert(
    html_string, html_to_markdown.ConversionOptions(extract_metadata=False)
).content

# Article-only extraction (strips nav/ads/boilerplate)
import trafilatura

article = trafilatura.extract(html_string, output_format="markdown", favor_recall=True)

When to Use Which