HasData
Cloud platform for extracting public web data. One API key, three execution modes. All endpoints sit under https://api.hasdata.com and authenticate with x-api-key.
curl -G 'https://api.hasdata.com/scrape/google/serp' \
--data-urlencode 'q=coffee' \
-H 'x-api-key: <your-api-key>'
401 invalid key, 403 quota exhausted, 429 concurrency cap, 500 server error (retry).
When to Use
Use this skill when:
- The user needs web scraping.
- The user needs search engine results.
- The user needs structured data extraction.
- The user needs ecommerce, travel, jobs, or local business data.
- The user explicitly asks about HasData.
Three execution modes
| Mode |
Latency |
When |
Endpoint |
| Web Scraping API |
seconds |
Arbitrary URL — JS rendering, CSS/AI extraction, screenshots |
POST /scrape/web |
| Scraper APIs (sync) |
seconds |
Pre-parsed JSON for known platforms (Google, Amazon, Zillow, …) |
GET /scrape/<vertical>/<resource> |
| Scraper Jobs (async) |
minutes–hours |
Bulk extraction, recursive crawling, webhook fan-out |
POST /scrapers/<slug>/jobs |
Decision rule. Default to a Scraper API when one exists for the platform (pre-parsed JSON, no selector maintenance). Use Web Scraping for arbitrary URLs not covered by an API. Reach for a Scraper Job only when no API equivalent exists — crawler, contacts, sec-edgar, amazon-bestsellers, amazon-product-reviews — or when async fan-out + webhooks save engineering time over a paginated client loop.
Always-true response shape
{ "requestMetadata": { "id": "…", "status": "ok", "url": "…" }, "...": "endpoint-specific" }
Treat data as valid only if requestMetadata.status === "ok". HTTP 200 alone isn't enough.
High-leverage patterns
- SERP-first enrichment. Google SERP can surface public snippets for company and professional-profile lookup. Use it for business or authorized research, avoid unnecessary direct scraping, and treat personal email/phone lookup as allowed only with a legitimate purpose and user authorization.
- AI Mode + verify.
/scrape/google/ai-mode for the answer + references → /scrape/web (markdown) on each reference URL → cited RAG context, no vector DB.
- Maps → leads.
/scrape/google-maps/search returns business websites and phones; collect contact details only from public, permitted sources and apply opt-out, rate, and privacy-law constraints before any outreach use.
- Crawler → corpus.
crawler Scraper Job with outputFormat: ["markdown"] + includePaths: "/docs/.+" produces an LLM-ready corpus in one submission.
- Pre-extracted via SERP rich snippets.
knowledgeGraph, localResults, inlineShoppingResults, relatedQuestions carry pre-parsed public facts. Always check them before considering direct page access.
When to call from code (the wiring)
- Auth:
x-api-key header on every request. Read from HASDATA_API_KEY env. Never hardcode, never log.
- Timeouts: set client timeout ≥ 300 s. HasData's own deadline is 300 s; shorter clients produce phantom failures while still being billed on completion.
- Retries:
429 and 5xx only — exponential backoff, jitter. Never retry 4xx (auth, validation).
- Concurrency: cap at your plan limit. The free tier is 1; anything higher just generates
429s.
- Async jobs: the submit response handle is
body.id (integer), not jobId. Persist it immediately. Poll GET /scrapers/jobs/<id> every 10–30 s with backoff; treat webhooks as best-effort and always pair with polling. On finished the status carries data: {csv, json, xlsx} short-lived URLs — download immediately.
See references/code-recipes.md for ready-to-paste Python and TypeScript clients with retry, backoff, bounded concurrency, and the full job lifecycle.
Common gotchas
- 300 s server deadline. Match client timeout.
- Disable
jsRendering first, enable only if the page needs it — most static pages parse fine without a headless browser.
- No
cookies parameter — cookies go through headers["Cookie"].
includePaths regex is case-sensitive. /blog/.+ won't match /Blog/....
- Scraper Job
data is double-wrapped. Each row is body.data[i].data; outer wraps with id, jobId, dataId, createdAt, updatedAt.
requestMetadata.status === "ok" is the only success signal. HTTP 200 alone isn't enough.
- Webhooks are best-effort with 3 retries. Always have a polling fallback.
References
references/web-scraping.md — POST /scrape/web parameters, JS scenarios, AI extraction, cookie auth.
references/search.md — Google SERP / Light / AI Mode / News / Shopping / Bing / Trends + pagination.
references/ecommerce.md — Amazon (product, search, seller, seller-products) and Shopify.
references/real-estate.md — Zillow, Redfin (bracketed filters).
references/travel.md — Airbnb, Booking, Google Flights (occupancy rules, token pagination, IATA codes).
references/local-business.md — Maps (search/place/reviews/photos/posts), Yelp, YellowPages.
references/jobs.md — Indeed and Glassdoor.
references/youtube.md — YouTube search / video / channel / transcript.
references/scraper-jobs.md — async submit/poll/results, Crawler, Contacts, SEC EDGAR, webhook receiver.
references/code-recipes.md — Python / TypeScript clients with retry, backoff, concurrency, polling.
Resources
Limitations
- Requires access to HasData services and valid credentials.
- Data quality and available fields depend on the target website and extraction method used.
- JavaScript-heavy websites may require rendering, which can affect performance and cost.
- Use only for public data or content the user is authorized to access; respect site terms, robots/access controls, privacy law, and rate limits.
- Rate limits, quotas, and account restrictions may apply depending on the endpoint and subscription plan.
1---2name: hasdata3description: "Use HasData APIs for web scraping and structured web data extraction."4version: 1.0.05author:6 name: "sickn33 / Antigravity Community"7 github: sickn338tags: [hasdata]9category: uncategorized10platforms: [macos, linux, windows]11license: MIT12metadata:13 hermes:14 source: https://skillmd.com/skills/nous-hermeshub/hasdata15 upstream: "antigravity-awesome-skills"16 risk: "safe"17 date_added: "2026-06-04"18 original_author: "Antigravity Community"19---2021# HasData2223Cloud platform for extracting public web data. One API key, three execution modes. All endpoints sit under `https://api.hasdata.com` and authenticate with `x-api-key`.2425```bash26curl -G 'https://api.hasdata.com/scrape/google/serp' \27 --data-urlencode 'q=coffee' \28 -H 'x-api-key: <your-api-key>'29```3031`401` invalid key, `403` quota exhausted, `429` concurrency cap, `500` server error (retry).3233## When to Use3435Use this skill when:3637- The user needs web scraping.38- The user needs search engine results.39- The user needs structured data extraction.40- The user needs ecommerce, travel, jobs, or local business data.41- The user explicitly asks about HasData.4243## Three execution modes4445| Mode | Latency | When | Endpoint |46|---|---|---|---|47| **Web Scraping API** | seconds | Arbitrary URL — JS rendering, CSS/AI extraction, screenshots | `POST /scrape/web` |48| **Scraper APIs** (sync) | seconds | Pre-parsed JSON for known platforms (Google, Amazon, Zillow, …) | `GET /scrape/<vertical>/<resource>` |49| **Scraper Jobs** (async) | minutes–hours | Bulk extraction, recursive crawling, webhook fan-out | `POST /scrapers/<slug>/jobs` |5051**Decision rule.** Default to a **Scraper API** when one exists for the platform (pre-parsed JSON, no selector maintenance). Use **Web Scraping** for arbitrary URLs not covered by an API. Reach for a **Scraper Job** only when no API equivalent exists — `crawler`, `contacts`, `sec-edgar`, `amazon-bestsellers`, `amazon-product-reviews` — *or* when async fan-out + webhooks save engineering time over a paginated client loop.5253## Always-true response shape5455```json56{ "requestMetadata": { "id": "…", "status": "ok", "url": "…" }, "...": "endpoint-specific" }57```5859Treat data as valid only if `requestMetadata.status === "ok"`. HTTP 200 alone isn't enough.6061## High-leverage patterns6263- **SERP-first enrichment.** Google SERP can surface public snippets for company and professional-profile lookup. Use it for business or authorized research, avoid unnecessary direct scraping, and treat personal email/phone lookup as allowed only with a legitimate purpose and user authorization.64- **AI Mode + verify.** `/scrape/google/ai-mode` for the answer + references → `/scrape/web` (markdown) on each reference URL → cited RAG context, no vector DB.65- **Maps → leads.** `/scrape/google-maps/search` returns business websites and phones; collect contact details only from public, permitted sources and apply opt-out, rate, and privacy-law constraints before any outreach use.66- **Crawler → corpus.** `crawler` Scraper Job with `outputFormat: ["markdown"]` + `includePaths: "/docs/.+"` produces an LLM-ready corpus in one submission.67- **Pre-extracted via SERP rich snippets.** `knowledgeGraph`, `localResults`, `inlineShoppingResults`, `relatedQuestions` carry pre-parsed public facts. Always check them before considering direct page access.6869## When to call from code (the wiring)7071- **Auth:** `x-api-key` header on every request. Read from `HASDATA_API_KEY` env. Never hardcode, never log.72- **Timeouts:** **set client timeout ≥ 300 s.** HasData's own deadline is 300 s; shorter clients produce phantom failures while still being billed on completion.73- **Retries:** `429` and `5xx` only — exponential backoff, jitter. Never retry `4xx` (auth, validation).74- **Concurrency:** cap at your plan limit. The free tier is 1; anything higher just generates `429`s.75- **Async jobs:** the submit response handle is `body.id` (integer), **not `jobId`**. Persist it immediately. Poll `GET /scrapers/jobs/<id>` every 10–30 s with backoff; treat webhooks as best-effort and always pair with polling. On `finished` the status carries `data: {csv, json, xlsx}` short-lived URLs — download immediately.7677See `references/code-recipes.md` for ready-to-paste Python and TypeScript clients with retry, backoff, bounded concurrency, and the full job lifecycle.7879## Common gotchas8081- **300 s server deadline.** Match client timeout.82- **Disable `jsRendering` first**, enable only if the page needs it — most static pages parse fine without a headless browser.83- **No `cookies` parameter** — cookies go through `headers["Cookie"]`.84- **`includePaths` regex is case-sensitive.** `/blog/.+` won't match `/Blog/...`.85- **Scraper Job `data` is double-wrapped.** Each row is `body.data[i].data`; outer wraps with `id`, `jobId`, `dataId`, `createdAt`, `updatedAt`.86- **`requestMetadata.status === "ok"` is the only success signal.** HTTP 200 alone isn't enough.87- **Webhooks are best-effort with 3 retries.** Always have a polling fallback.8889## References9091- [`references/web-scraping.md`](references/web-scraping.md) — `POST /scrape/web` parameters, JS scenarios, AI extraction, cookie auth.92- [`references/search.md`](references/search.md) — Google SERP / Light / AI Mode / News / Shopping / Bing / Trends + pagination.93- [`references/ecommerce.md`](references/ecommerce.md) — Amazon (product, search, seller, seller-products) and Shopify.94- [`references/real-estate.md`](references/real-estate.md) — Zillow, Redfin (bracketed filters).95- [`references/travel.md`](references/travel.md) — Airbnb, Booking, Google Flights (occupancy rules, token pagination, IATA codes).96- [`references/local-business.md`](references/local-business.md) — Maps (search/place/reviews/photos/posts), Yelp, YellowPages.97- [`references/jobs.md`](references/jobs.md) — Indeed and Glassdoor.98- [`references/youtube.md`](references/youtube.md) — YouTube search / video / channel / transcript.99- [`references/scraper-jobs.md`](references/scraper-jobs.md) — async submit/poll/results, Crawler, Contacts, SEC EDGAR, webhook receiver.100- [`references/code-recipes.md`](references/code-recipes.md) — Python / TypeScript clients with retry, backoff, concurrency, polling.101102## Resources103104- Sitemap: <https://docs.hasdata.com/llms.txt>105- API status codes: <https://docs.hasdata.com/api-codes>106- Credits & concurrency: <https://docs.hasdata.com/credits-and-concurrency>107- Dashboard: <https://app.hasdata.com>108109## Limitations110111* Requires access to HasData services and valid credentials.112* Data quality and available fields depend on the target website and extraction method used.113* JavaScript-heavy websites may require rendering, which can affect performance and cost.114* Use only for public data or content the user is authorized to access; respect site terms, robots/access controls, privacy law, and rate limits.115* Rate limits, quotas, and account restrictions may apply depending on the endpoint and subscription plan.