Nimble Crawl Reference

nimble crawl — reference

Async bulk crawling — starts a crawl job, returns a crawl_id, then you poll for status and retrieve per-page content via individual task_ids.

Tip: For LLM use, prefer nimble map + nimble extract --format markdown (faster, cleaner). Use crawl when you need raw HTML archives of many pages at once.

Table of Contents

• 1. Create crawl (async)
• 2. Get crawl status
• 3. List crawls
• 4. Cancel crawl
• Polling guidelines
• Retrieving page content
• Crawl vs Map

1. Create crawl (async)

Parameters:

CLI:

nimble crawl run --url "https://docs.example.com" --limit 50 --name "docs-crawl"

Python SDK:

from nimble_python import Nimble
nimble = Nimble(api_key=os.environ["NIMBLE_API_KEY"])

resp = nimble.crawl.run(url="https://docs.example.com", limit=50, name="docs-crawl")
crawl_id = resp.crawl_id

Response fields: crawl_id, status (initial: queued), url, name, crawl_options, created_at

Crawl status values: queued → running → succeeded / failed / canceled

2. Get crawl status

Parameters:

CLI:

nimble crawl status --id "abc-123"

Python SDK:

crawl = nimble.crawl.status(id=crawl_id)
print(crawl.status, crawl.completed, crawl.total)

Response adds: total, pending, completed, failed, tasks[]

CRITICAL: Use tasks[].task_id — NOT crawl_id — to fetch page content. Using crawl_id with tasks returns 404.

3. List crawls

Parameters:

CLI:

nimble crawl list --status running

Python SDK:

result = nimble.crawl.list()
# result.data = list of crawl objects
# result.pagination = { has_next, next_cursor, total }

4. Cancel crawl

Parameters:

CLI:

nimble crawl terminate --id "abc-123"

Python SDK:

nimble.crawl.terminate(id=crawl_id)
# Returns: {"status": "canceled"}

Polling guidelines

CRAWL_ID=$(nimble crawl run --url "https://example.com" --limit 100 --name "my-crawl" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['crawl_id'])")

while true; do
  STATUS=$(nimble crawl status --id "$CRAWL_ID" \
    | python3 -c "import json,sys; print(json.load(sys.stdin).get('status',''))")
  echo "Status: $STATUS"
  [ "$STATUS" = "succeeded" ] || [ "$STATUS" = "failed" ] || [ "$STATUS" = "canceled" ] && break
  sleep 30
done

Retrieving page content

After crawl succeeded, fetch each completed task — see nimble-tasks reference:

nimble tasks results --task-id "task-456"

Crawl vs Map