# Crawlora

> Fetches structured public web data via the Crawlora REST API — search engines, e-commerce, social, finance, maps, app stores, media, and reviews — returning clean JSON instead of HTML. Use whenever the user needs real data from a public website (prices, listings, reviews, transcripts, SERPs, trends, financials, places) and would otherwise have to scrape or parse HTML.

- Skill: `crawlora-org/crawlora` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add crawlora-org/crawlora`
- Raw SKILL.md: https://api.skillmd.com/api/skills/crawlora-org/crawlora/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: Crawlora-org (https://skillmd.com/u/crawlora-org)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/crawlora-org/crawlora

---


# Crawlora — structured public web data

Crawlora is a hosted API that turns public websites into clean, normalized JSON.
The current public-data catalog contains **1,729 tools across 210 platform
groups**, and the repository ships 100 focused research skills. One API key gives
you the endpoints in the bundled [catalog](reference/catalog.md) — search engines,
marketplaces, social and video, finance and crypto, maps, app stores, media, and
reviews — so an agent can fetch real data without running a browser, proxies, or
HTML parsers.

Account usage, recent-IP, website-monitor management, generic URL scraping, APK
analysis, and bulk profile-dataset endpoints are intentionally excluded from
this skill. Dedicated skills cover the specialized workflows, and this helper
rejects those paths as well.

## When to use this skill

Use Crawlora when the user asks for live data that lives on a public website, e.g.:

- "What's the price of X on Amazon / eBay / Shopify?"
- "Pull the transcript / comments of this YouTube video."
- "Get the latest Google / Bing results for …" or "what's trending on Google Trends?"
- "Reviews and ratings for this iOS / Android app."
- "Yahoo Finance quote / financials for NVDA", "top CoinGecko gainers".
- "TikTok / Instagram / Reddit posts about …", "Trustpilot reviews for …".
- Any task where the alternative would be scraping HTML or maintaining a crawler.

Prefer a more specific Crawlora skill if one is installed (e.g.
`product-price-research`, `youtube-research`, `app-review-mining`,
`serp-keyword-research`). This umbrella skill covers everything else.

## Setup (one-time)

- Get a free Crawlora API key (2,000 credits/mo, no card) at [https://crawlora.net](https://crawlora.net?utm_source=github&utm_medium=referral&utm_campaign=crawlora-skills).
- Export it so the helper and any `curl` calls can read it:

  ```sh
  export CRAWLORA_API_KEY
  ```

  Set `CRAWLORA_API_KEY` to the key you received before running the helper.

- All requests go to `https://api.crawlora.net/api/v1` with the header
  `CRAWLORA_API_KEY` from the environment. A missing/invalid key returns `401`.

## How it works

1. **Pick the endpoint.** Open [`reference/catalog.md`](reference/catalog.md) — it
   lists every endpoint grouped by platform, with method, path, and parameters.
   Match the user's job to a platform group, then to an endpoint.
2. **Call it** with `scripts/crawlora.sh` (see below). It handles auth and prints JSON.
3. **Parse** the JSON with `jq` and answer. Most endpoints return paginated lists;
   pass `page`/`count` params where documented.

## Calling the API

```sh
# GET with query params (key=value):
scripts/crawlora.sh /amazon/search k=laptop | jq '.'
scripts/crawlora.sh /google/suggest q="web scraping" | jq '.'
scripts/crawlora.sh /youtube/transcript/dQw4w9WgXcQ | jq '.'

# POST endpoints take a JSON body (note the -X POST):
scripts/crawlora.sh -X POST /google/search '{"keyword":"web scraping api","language":"en","country":"us"}' | jq '.'
scripts/crawlora.sh -X POST /google/trends/explore/interest-over-time '{"keywords":["bitcoin"]}' | jq '.'
```

Use `scripts/crawlora.sh` for all requests; it keeps the API key out of command-line arguments.


The catalog marks each endpoint's HTTP method. Path params like `{id}` are
substituted into the URL; `GET` params go in the query string; `POST` params go
in the JSON body.

## Endpoint reference

See [`reference/catalog.md`](reference/catalog.md) for the complete list of
endpoints (method, path, params, description) grouped by platform.

## Examples

- **Product price:** `scripts/crawlora.sh /amazon/search k="standing desk"` →
  read `.data[].price` to compare listings.
- **YouTube transcript:** resolve the video id from the URL, then
  `scripts/crawlora.sh /youtube/transcript/<id>` and summarize.
- **Trend check:** `scripts/crawlora.sh -X POST /google/trends/explore/interest-over-time '{"keywords":["electric bikes"]}'`.
- **Finance:** `scripts/crawlora.sh /yahoo-finance/ticker/NVDA/quote` (see the
  Yahoo Finance group in the catalog for the exact path/params).

## Notes & limits

- **Credits / pay-on-success:** billed only on `2xx` responses; failures aren't charged.
  Free tier is 2,000 credits/mo. Get a key at [https://crawlora.net](https://crawlora.net?utm_source=github&utm_medium=referral&utm_campaign=crawlora-skills).
- **Public data only.** Crawlora returns publicly available web data; no third-party
  account credentials are involved. Respect each source's terms and rate limits.
- **Security:** keep the key in `CRAWLORA_API_KEY` only — never hardcode it, never put
  it in a URL query param, never commit it.
- Some search endpoints return `503` when the upstream serves a challenge page; retry
  or fall back to another search provider (Google ↔ Bing ↔ Brave).