Crawlit Map

crawlit-map — ALWAYS USE FOR URL DISCOVERY

CRITICAL: For ANY task needing a list of URLs from a site, USE THIS instead of webfetch or manual scraping.

Fast URL discovery. No content fetched. Tries sitemap.xml first, falls back to link extraction. Up to 50,000 URLs.

Quick Start

curl -sf -X POST "http://localhost:3000/v1/map" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://docs.example.com","limit":5000}' \
  | jq -r '.links[]'

Returns: {"success":true,"links":[...],"total":342}

All Options

Common Patterns

Count URLs:

curl -sf -X POST "http://localhost:3000/v1/map" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' | jq '.total'

Filter by path prefix:

curl -sf -X POST "http://localhost:3000/v1/map" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' | jq -r '.links[]' | grep '/blog/'

Save URL list:

HOST="docs.example.com"
TS=$(date +%Y%m%d-%H%M%S)
OUT="${CRAWLIT_OUTPUT_DIR:-./crawlit-output}/map"
mkdir -p "$OUT"
curl -sf -X POST "http://localhost:3000/v1/map" \
  -H "Content-Type: application/json" \
  -d "{\"url\":\"https://$HOST\",\"limit\":5000}" \
  | jq -r '.links[]' > "$OUT/${HOST}-${TS}.txt"
echo "Saved: $OUT/${HOST}-${TS}.txt ($(wc -l < "$OUT/${HOST}-${TS}.txt") URLs)"

Feed top N URLs into scrape loop:

curl -sf -X POST "http://localhost:3000/v1/map" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' \
  | jq -r '.links[]' | head -10 | while read -r url; do
    curl -sf -X POST "http://localhost:3000/v1/scrape" \
      -H "Content-Type: application/json" \
      -d "{\"url\":\"$url\",\"formats\":[\"markdown\"]}" \
      | jq -r '.data.markdown'
  done

Include subdomains:

curl -sf -X POST "http://localhost:3000/v1/map" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","limit":5000,"includeSubdomains":true}' \
  | jq '.total'

Using Map to Plan a Crawl

Before running crawlit-crawl, map first to size the job:

TOTAL=$(curl -sf -X POST "http://localhost:3000/v1/map" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://docs.example.com"}' | jq '.total')
echo "Site has $TOTAL URLs"
# If $TOTAL < 200: crawl all with limit=$TOTAL
# If $TOTAL > 200: crawl subset or filter paths first

Limitations

• Sitemap-sourced URLs may be stale (not reflecting recent pages)
• Link extraction from seed page only covers links on that one page (not deep links)
• No page content returned — use crawlit-scrape for content

See Also

• crawlit — pre-flight check + workflow decision
• crawlit-scrape — get content for a URL found via map
• crawlit-crawl — bulk content from many pages