Navinspire-ia Scrape Operator
Overview
Deliver corpora from any site type into workspace files (csv/json/xlsx/md/report). When writing Python scrapers, use Scrapling 0.4.14 first (see skill scrapling). Use the scrape tool (Rust / httpx) for one-shot jobs with no custom code. Use browser for JS/scroll/forms. Finish with files + a one-line done summary.
Routing by page type
| Site type | Approach |
|---|---|
| Custom scraper / spider the agent writes | Scrapling 0.4.14 (scrapling skill) |
| Static docs / blogs, no custom code | scrape crawl or pipeline (Rust when navin-core is built) |
| Listing + next/page=N | Scrapling spider, or scrape paginate then crawl item links |
| Sitemap known | Scrapling or sitemap → fetch urls in batches → export |
| Infinite scroll / lazy JS | browser navigate → scroll_infinite → extract / content → hand off to export |
| Forms / multi-tab / upload | browser (see playwright-browser) |
| empty_shell after fetch | escalate browser |
| captcha / CF / paywall / login | pause - never bypass |
Mass crawl checklist
diagnosea seed URL.- Cap with
max_pages(hard max 500) and optionalallow/denyregex. - Resume long jobs with
checkpoint="scrape/checkpoint.json". - Prefer
pipelineto write exports directly; keep chat short. - Dedup is on by content hash; pagination links do not burn depth budget.
Cheat sheet
# Custom code: Scrapling 0.4.14 (Fetcher / StealthyFetcher / Spider / ShopifySpider)
# see skill scrapling and scrapling/references/api.md
scrape(action="paginate", url="https://shop.example/list?page=1", max_pages=30)
scrape(action="crawl", url="https://shop.example/list", max_depth=2, max_pages=200, checkpoint="scrape/ckpt.json", deny="/(cart|login|logout)")
scrape(action="sitemap", url="https://example.com/sitemap.xml", max_pages=200)
scrape(action="pipeline", url="https://example.com", format="xlsx", path="scrape/out.xlsx", max_pages=100)
browser(action="scroll_infinite", index=12) # after navigate - then extract/content
Rules
- Developing scraper code: Scrapling 0.4.14 first. Do not start with BeautifulSoup or raw Playwright.
- One-shot corpora without new code:
scrapetool (Rust hot path, else httpx). - Escalate to
browserfor scroll/JS/forms only. - Respect robots.txt (default on). Never invent rows.
- Never bypass human walls. Write files, not chat dumps.
- Deliver the dataset and the report. Scraper code stays under
scrape/build/and is handed over only when asked for. - Qualified people/companies from a scrape can be written into the shared Studio CRM with the
crmtool after a quality pass.