Crawlora

Crawlora — structured public web data

Crawlora is a hosted API that turns public websites into clean, normalized JSON. The current public-data catalog contains 1,729 tools across 210 platform groups, and the repository ships 100 focused research skills. One API key gives you the endpoints in the bundled catalog — search engines, marketplaces, social and video, finance and crypto, maps, app stores, media, and reviews — so an agent can fetch real data without running a browser, proxies, or HTML parsers.

Account usage, recent-IP, website-monitor management, generic URL scraping, APK analysis, and bulk profile-dataset endpoints are intentionally excluded from this skill. Dedicated skills cover the specialized workflows, and this helper rejects those paths as well.

When to use this skill

Use Crawlora when the user asks for live data that lives on a public website, e.g.:

• "What's the price of X on Amazon / eBay / Shopify?"
• "Pull the transcript / comments of this YouTube video."
• "Get the latest Google / Bing results for …" or "what's trending on Google Trends?"
• "Reviews and ratings for this iOS / Android app."
• "Yahoo Finance quote / financials for NVDA", "top CoinGecko gainers".
• "TikTok / Instagram / Reddit posts about …", "Trustpilot reviews for …".
• Any task where the alternative would be scraping HTML or maintaining a crawler.

Prefer a more specific Crawlora skill if one is installed (e.g. product-price-research, youtube-research, app-review-mining, serp-keyword-research). This umbrella skill covers everything else.

Setup (one-time)

• Get a free Crawlora API key (2,000 credits/mo, no card) at https://crawlora.net.

• Export it so the helper and any curl calls can read it:

  export CRAWLORA_API_KEY
  

  Set CRAWLORA_API_KEY to the key you received before running the helper.

• All requests go to https://api.crawlora.net/api/v1 with the header CRAWLORA_API_KEY from the environment. A missing/invalid key returns 401.

How it works

1. Pick the endpoint. Open reference/catalog.md — it lists every endpoint grouped by platform, with method, path, and parameters. Match the user's job to a platform group, then to an endpoint.
2. Call it with scripts/crawlora.sh (see below). It handles auth and prints JSON.
3. Parse the JSON with jq and answer. Most endpoints return paginated lists; pass page/count params where documented.

Calling the API

# GET with query params (key=value):
scripts/crawlora.sh /amazon/search k=laptop | jq '.'
scripts/crawlora.sh /google/suggest q="web scraping" | jq '.'
scripts/crawlora.sh /youtube/transcript/dQw4w9WgXcQ | jq '.'

# POST endpoints take a JSON body (note the -X POST):
scripts/crawlora.sh -X POST /google/search '{"keyword":"web scraping api","language":"en","country":"us"}' | jq '.'
scripts/crawlora.sh -X POST /google/trends/explore/interest-over-time '{"keywords":["bitcoin"]}' | jq '.'

Use scripts/crawlora.sh for all requests; it keeps the API key out of command-line arguments.

The catalog marks each endpoint's HTTP method. Path params like {id} are substituted into the URL; GET params go in the query string; POST params go in the JSON body.

Endpoint reference

See reference/catalog.md for the complete list of endpoints (method, path, params, description) grouped by platform.

Examples

• Product price: scripts/crawlora.sh /amazon/search k="standing desk" → read .data[].price to compare listings.
• YouTube transcript: resolve the video id from the URL, then scripts/crawlora.sh /youtube/transcript/<id> and summarize.
• Trend check: scripts/crawlora.sh -X POST /google/trends/explore/interest-over-time '{"keywords":["electric bikes"]}'.
• Finance: scripts/crawlora.sh /yahoo-finance/ticker/NVDA/quote (see the Yahoo Finance group in the catalog for the exact path/params).

Notes & limits

• Credits / pay-on-success: billed only on 2xx responses; failures aren't charged. Free tier is 2,000 credits/mo. Get a key at https://crawlora.net.
• Public data only. Crawlora returns publicly available web data; no third-party account credentials are involved. Respect each source's terms and rate limits.
• Security: keep the key in CRAWLORA_API_KEY only — never hardcode it, never put it in a URL query param, never commit it.
• Some search endpoints return 503 when the upstream serves a challenge page; retry or fall back to another search provider (Google ↔ Bing ↔ Brave).