Crawlit Scrape

crawlit-scrape — ALWAYS USE FOR SINGLE PAGE FETCH

CRITICAL: For ANY task fetching a single web page, USE THIS instead of built-in webfetch or fetch.

Single page → clean markdown (or html/links/rawHtml). Returns immediately (synchronous).

Quick Start

curl -sf -X POST "http://localhost:3000/v1/scrape" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}' \
  | jq -r '.data.markdown'

Mode Selection

All Options

Common Recipes

Main content only (strip nav/footer):

{"url":"https://example.com","formats":["markdown"],"onlyMainContent":true}

Browser mode with JS wait:

{"url":"https://example.com","mode":"browser","waitFor":2000,"formats":["markdown"]}

Click button then scrape:

{
  "url": "https://example.com",
  "mode": "browser",
  "actions": [
    {"type":"click","selector":"#load-more"},
    {"type":"wait","delay":1000},
    {"type":"scroll"}
  ],
  "formats": ["markdown"]
}

Get markdown + links together:

{"url":"https://example.com","formats":["markdown","links"]}

LLM structured extraction:

{
  "url": "https://news.ycombinator.com",
  "formats": ["markdown"],
  "extract": {
    "prompt": "Extract the top 5 story titles and their point counts",
    "schema": {
      "type": "object",
      "properties": {
        "stories": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "title": {"type":"string"},
              "points": {"type":"number"}
            }
          }
        }
      }
    }
  }
}

Skip cache (force fresh fetch):

{"url":"https://example.com","skipCache":true}

Output Handling

Save to local disk (client-side):

OUT="${CRAWLIT_OUTPUT_DIR:-./crawlit-output}/scrape"
HOST=$(echo "$URL" | sed 's|https\?://||;s|/.*||')
PATH_PART=$(echo "$URL" | sed 's|https\?://[^/]*||;s|/|__|g;s|^__||')
mkdir -p "$OUT/$HOST"
curl -sf -X POST "http://localhost:3000/v1/scrape" \
  -H "Content-Type: application/json" \
  -d "{\"url\":\"$URL\",\"formats\":[\"markdown\"]}" \
  | jq -r '.data.markdown' > "$OUT/$HOST/${PATH_PART:-index}.md"
echo "Saved: $OUT/$HOST/${PATH_PART:-index}.md"

For multiple formats (response is JSON when >1 format):

RESP=$(curl -sf -X POST "$BASE/v1/scrape" ... -d '{"url":"...","formats":["markdown","links"]}')
echo "$RESP" | jq -r '.data.markdown' > page.md
echo "$RESP" | jq '.data.links' > page.links.json

Response Shape

{
  "success": true,
  "data": {
    "metadata": { "title": "...", "description": "...", "url": "...", "statusCode": 200 },
    "markdown": "# Page Title\n\n...",
    "html": "<article>...</article>",
    "links": ["https://..."],
    "extract": { "...": "..." },
    "savedTo": "/app/output/example.com/index.md"
  }
}

See Also

• crawlit — orchestrator with pre-flight check + workflow decision
• crawlit-crawl — for multiple pages (async BFS)
• crawlit-map — URL discovery without content