Scraping

URL Scraping

This skill covers scraping single URLs with Firecrawl — output formats, content selection, wait strategies, and authentication.

Available Tools

Basic Scraping

Tool: scrape_url
Input: {
  "url": "https://example.com/page",
  "formats": ["markdown"]
}

Returns: Page content in specified format with metadata (title, description, status code).

Output Formats

Multiple Formats

Tool: scrape_url
Input: {
  "url": "https://example.com",
  "formats": ["markdown", "links"]
}

Content Selection

Include Only Specific Elements

Tool: scrape_url
Input: {
  "url": "https://example.com/article",
  "formats": ["markdown"],
  "includeTags": ["article", "main", ".content"]
}

Exclude Elements

Tool: scrape_url
Input: {
  "url": "https://example.com/article",
  "formats": ["markdown"],
  "excludeTags": ["nav", "footer", ".sidebar", ".ads"]
}

Advanced Parameters

Main Content Only

Tool: scrape_url
Input: {
  "url": "https://example.com/article",
  "formats": ["markdown"],
  "onlyMainContent": true
}

Mobile Rendering

Tool: scrape_url
Input: {
  "url": "https://example.com/page",
  "formats": ["markdown"],
  "mobile": true
}

LLM-Powered Extraction During Scrape

Tool: scrape_url
Input: {
  "url": "https://example.com/pricing",
  "formats": ["markdown"],
  "prompt": "Extract the pricing information and plan details"
}

Structured Extraction During Scrape

Tool: scrape_url
Input: {
  "url": "https://example.com/product",
  "formats": ["markdown"],
  "schema": {
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "price": { "type": "number" }
    }
  }
}

Wait Strategies

For pages with dynamic content (JavaScript-rendered):

Wait for Selector

Tool: scrape_url
Input: {
  "url": "https://example.com/spa",
  "formats": ["markdown"],
  "waitFor": ".product-list"
}

Wait for Fixed Time (ms)

Tool: scrape_url
Input: {
  "url": "https://example.com/slow-page",
  "formats": ["markdown"],
  "timeout": 30000
}

Headers and Cookies

Custom Headers

Tool: scrape_url
Input: {
  "url": "https://example.com/api-docs",
  "formats": ["markdown"],
  "headers": {
    "Authorization": "Bearer token123",
    "Accept-Language": "en-US"
  }
}

Common Workflows

Scrape and Analyze Content

1. scrape_url(url, formats=["markdown"]) -> Get clean text
2. Analyze content (summarize, extract key points, etc.)

Scrape with Link Discovery

1. scrape_url(url, formats=["markdown", "links"]) -> Get content + all links
2. Filter links matching pattern
3. Scrape specific linked pages

Scrape Authenticated Page

1. scrape_url(url, headers={Authorization: "Bearer ..."}) -> Get protected content

Error Handling Patterns

Common Failure Scenarios

• URL unreachable → verify URL is accessible; check for typos, DNS issues, or geo-blocking
• Timeout → increase timeout parameter, or switch to async batch scrape for slow pages
• Rate limited (429) → add delays between requests; reduce concurrency
• Content blocked → try mobile: true for mobile rendering, or set custom headers (User-Agent)
• Empty content → page may require JavaScript; use waitFor with a CSS selector that appears after load
• Authentication required → use headers with auth tokens, or use browser sessions for login flows

Defensive Workflow

1. Start with a single scrape_url to test if the page is accessible
2. If empty content: add waitFor parameter with a page-specific selector
3. If blocked: try with mobile: true or custom User-Agent header
4. For many URLs: switch to batch_scrape (see batch-operations skill)
5. For authenticated pages: use browser sessions (see browser-sessions skill)

Best Practices

1. Use markdown format by default — cleanest output for analysis
2. Exclude nav/footer — focus on main content with excludeTags
3. Use waitFor for SPAs — dynamic pages need time to render
4. Check status codes — verify page loaded successfully
5. Respect rate limits — don't scrape too many pages too fast
6. Use includeTags for precision — target specific page sections