HasData
Cloud platform for extracting public web data. One API key, three execution modes. All endpoints sit under https://api.hasdata.com and authenticate with the x-api-key header.
When to Use
Use this skill when:
- The user needs web scraping of arbitrary URLs.
- The user needs search engine results (Google, Bing, Trends).
- The user needs structured data extraction (ecommerce, real estate, travel, jobs, local business, YouTube).
- The user needs bulk/async crawling with webhook fan-out.
- The user explicitly asks about HasData or
api.hasdata.com.
Prerequisites
A valid HasData API key. Store it in the environment variable HASDATA_API_KEY. Never hardcode, never log.
On Windows (PowerShell), set it for the current session:
$env:HASDATA_API_KEY = "YOUR_KEY"
Or persist it for future sessions:
[Environment]::SetEnvironmentVariable("HASDATA_API_KEY", "YOUR_KEY", "User")
HTTP client with a configurable timeout of at least 300 seconds (see Pitfalls).
Network egress to https://api.hasdata.com.
Three Execution Modes
| Mode |
Latency |
When |
Endpoint |
| Web Scraping API |
seconds |
Arbitrary URL — JS rendering, CSS/AI extraction, screenshots |
POST /scrape/web |
| Scraper APIs (sync) |
seconds |
Pre-parsed JSON for known platforms (Google, Amazon, Zillow, …) |
GET /scrape/<vertical>/<resource> |
| Scraper Jobs (async) |
minutes–hours |
Bulk extraction, recursive crawling, webhook fan-out |
POST /scrapers/<slug>/jobs |
Decision rule. Default to a Scraper API when one exists for the target platform (pre-parsed JSON, no selector maintenance). Use Web Scraping for arbitrary URLs not covered by an API. Reach for a Scraper Job only when no API equivalent exists — crawler, contacts, sec-edgar, amazon-bestsellers, amazon-product-reviews — or when async fan-out + webhooks save engineering time over a paginated client loop.
Always-True Response Shape
{
"requestMetadata": { "id": "…", "status": "ok", "url": "…" },
"...": "endpoint-specific"
}
HARD RULE: Treat data as valid only if requestMetadata.status === "ok". HTTP 200 alone is not a success signal.
Procedure
1. Quick smoke test (Google SERP)
curl -G 'https://api.hasdata.com/scrape/google/serp' `
--data-urlencode 'q=coffee' `
-H 'x-api-key: YOUR_KEY'
Expected: JSON with requestMetadata.status equal to "ok".
2. Choose the execution mode
- Scraper API — if the platform has a dedicated endpoint (Google, Amazon, Zillow, Airbnb, Indeed, YouTube, etc.). Load the matching reference file (see Reference Loading Guide below) for parameters.
- Web Scraping API — for arbitrary URLs. Load
references/web-scraping.md.
- Scraper Job — for bulk/async. Load
references/scraper-jobs.md.
3. Web Scraping API (POST /scrape/web)
Load references/web-scraping.md before constructing the request.
curl -X POST 'https://api.hasdata.com/scrape/web' `
-H 'x-api-key: YOUR_KEY' `
-H 'Content-Type: application/json' `
-d '{\"url\":\"https://example.com\",\"jsRendering\":false,\"outputFormat\":\"markdown\"}'
Key parameters (full list in reference):
url — target URL (required).
jsRendering — default to false. Enable only if the page requires a headless browser.
outputFormat — markdown, html, text, or screenshot.
extract — CSS selectors or AI extraction prompt.
headers — pass custom headers including Cookie (there is no top-level cookies parameter).
4. Scraper APIs (sync, GET /scrape/<vertical>/<resource>)
Load the relevant reference file for the vertical:
curl -G 'https://api.hasdata.com/scrape/google/serp' `
--data-urlencode 'q=coffee' `
--data-urlencode 'num=10' `
-H 'x-api-key: YOUR_KEY'
Check requestMetadata.status === "ok" before consuming results. Inspect rich-snippet fields (knowledgeGraph, localResults, inlineShoppingResults, relatedQuestions) before considering direct page access.
5. Scraper Jobs (async, POST /scrapers/<slug>/jobs)
Load references/scraper-jobs.md before constructing the request.
curl -X POST 'https://api.hasdata.com/scrapers/crawler/jobs' `
-H 'x-api-key: YOUR_KEY' `
-H 'Content-Type: application/json' `
-d '{\"url\":\"https://example.com\",\"outputFormat\":[\"markdown\"],\"includePaths\":\"/docs/.+\"}'
HARD RULES for async jobs:
- The submit response handle is
body.id (an integer), not jobId. Persist it immediately.
- Poll
GET /scrapers/jobs/<id> every 10–30 s with backoff.
- Treat webhooks as best-effort (3 retries). Always pair with polling.
- On
finished, the status carries data: {csv, json, xlsx} short-lived URLs — download immediately.
6. High-leverage patterns
- SERP-first enrichment. Use Google SERP to surface public snippets for company/professional lookup before attempting direct scraping. Treat personal email/phone lookup as allowed only with a legitimate purpose and user authorization.
- AI Mode + verify.
/scrape/google/ai-mode for the answer + references → /scrape/web (markdown) on each reference URL → cited RAG context, no vector DB.
- Maps → leads.
/scrape/google-maps/search returns business websites and phones. Apply opt-out, rate, and privacy-law constraints before any outreach use.
- Crawler → corpus.
crawler Scraper Job with outputFormat: ["markdown"] + includePaths: "/docs/.+" produces an LLM-ready corpus in one submission.
- Pre-extracted via SERP rich snippets.
knowledgeGraph, localResults, inlineShoppingResults, relatedQuestions carry pre-parsed public facts. Always check them before considering direct page access.
Reference Loading Guide
Load these files on demand — when the user's request maps to a specific vertical or pattern:
| Reference file |
Load when… |
references/web-scraping.md |
Using POST /scrape/web — parameters, JS scenarios, AI extraction, cookie auth. |
references/search.md |
Google SERP / Light / AI Mode / News / Shopping / Bing / Trends + pagination. |
references/ecommerce.md |
Amazon (product, search, seller, seller-products) and Shopify. |
references/real-estate.md |
Zillow, Redfin (bracketed filters). |
references/travel.md |
Airbnb, Booking, Google Flights (occupancy rules, token pagination, IATA codes). |
references/local-business.md |
Maps (search/place/reviews/photos/posts), Yelp, YellowPages. |
references/jobs.md |
Indeed and Glassdoor. |
references/youtube.md |
YouTube search / video / channel / transcript. |
references/scraper-jobs.md |
Async submit/poll/results, Crawler, Contacts, SEC EDGAR, webhook receiver. |
references/code-recipes.md |
Ready-to-paste Python and TypeScript clients with retry, backoff, bounded concurrency, and the full job lifecycle. |
Wiring from Code
- Auth:
x-api-key header on every request. Read from HASDATA_API_KEY env. Never hardcode, never log.
- Timeouts: Set client timeout ≥ 300 s. HasData's own deadline is 300 s; shorter clients produce phantom failures while still being billed on completion.
- Retries:
429 and 5xx only — exponential backoff with jitter. Never retry 4xx (auth, validation).
- Concurrency: Cap at your plan limit. The free tier is 1; anything higher just generates
429s.
- Async jobs: Handle is
body.id (integer), not jobId. Poll GET /scrapers/jobs/<id> every 10–30 s. Webhooks are best-effort; always pair with polling. Download result URLs immediately on finished.
See references/code-recipes.md for complete Python and TypeScript client implementations.
Pitfalls
- 300 s server deadline. Match your client timeout to at least 300 s. Shorter timeouts cause phantom failures while still being billed.
requestMetadata.status === "ok" is the only success signal. HTTP 200 alone is not enough — always check the metadata status field.
- Disable
jsRendering first. Enable only if the page needs it — most static pages parse fine without a headless browser. Rendering costs more and is slower.
- No
cookies parameter. Cookies go through headers["Cookie"], not a top-level field.
includePaths regex is case-sensitive. /blog/.+ will not match /Blog/....
- Scraper Job
data is double-wrapped. Each row is body.data[i].data; the outer wrapper carries id, jobId, dataId, createdAt, updatedAt.
- Async job handle is
body.id (integer), not jobId. Persist it immediately after submit.
- Webhooks are best-effort with 3 retries. Always have a polling fallback.
- Concurrency cap. Free tier is 1 concurrent request. Exceeding your plan limit generates
429s.
- Never retry
4xx. 401 = invalid key, 403 = quota exhausted, 429 = concurrency cap. Only 429 and 5xx are retryable.
- Result URLs are short-lived. Download
csv/json/xlsx URLs immediately on job finished.
Verification
API key validity — run the smoke test and confirm requestMetadata.status === "ok":
curl -G 'https://api.hasdata.com/scrape/google/serp' `
--data-urlencode 'q=test' `
-H "x-api-key: $env:HASDATA_API_KEY"
Expected output contains: "status": "ok".
Error code mapping — verify you handle each correctly:
| HTTP |
Meaning |
Retry? |
401 |
Invalid key |
No |
403 |
Quota exhausted |
No |
429 |
Concurrency cap |
Yes (backoff) |
500 |
Server error |
Yes (backoff) |
Async job lifecycle — after POST /scrapers/<slug>/jobs, verify:
- Response contains
id (integer) — not jobId.
GET /scrapers/jobs/<id> eventually returns status finished.
data.csv, data.json, or data.xlsx URLs are present and downloadable.
Client timeout — confirm your HTTP client timeout is set to ≥ 300 s.
Resources
Limitations
- Requires access to HasData services and valid credentials.
- Data quality and available fields depend on the target website and extraction method used.
- JavaScript-heavy websites may require rendering, which can affect performance and cost.
- Use only for public data or content the user is authorized to access; respect site terms, robots/access controls, privacy law, and rate limits.
- Rate limits, quotas, and account restrictions may apply depending on the endpoint and subscription plan.
1---
2name: hasdata
3description: "Calls HasData (api.hasdata.com, x-api-key) for Google/Bing SERP JSON, vertical scrapers (Amazon, Zillow, Maps), POST /scrape/web on arbitrary public URLs, and async Scraper Jobs with poll plus webhook. Use when the request is HasData, public SERP, or bulk public crawl. Not for authenticated first-party product APIs, CAPTCHA bypass, or Playwright/CDP sessions that never hit HasData."
4version: 1.0.1
5risk: safe
6source: official
7source_type: official
8source_repo: HasData/hasdata-cli
9license: MIT
10license_source: "https://github.com/HasData/hasdata-cli/blob/main/LICENSE"
11date_added: "2026-06-04"
12metadata:
13 source: https://skillmd.com/skills/kayforkind/hasdata
14---
15
16# HasData
17
18Cloud platform for extracting public web data. One API key, three execution modes. All endpoints sit under `https://api.hasdata.com` and authenticate with the `x-api-key` header.
19
20## When to Use
21
22Use this skill when:
23
24- The user needs web scraping of arbitrary URLs.
25- The user needs search engine results (Google, Bing, Trends).
26- The user needs structured data extraction (ecommerce, real estate, travel, jobs, local business, YouTube).
27- The user needs bulk/async crawling with webhook fan-out.
28- The user explicitly asks about HasData or `api.hasdata.com`.
29
30## Prerequisites
31
32- A valid HasData API key. Store it in the environment variable `HASDATA_API_KEY`. Never hardcode, never log.
33- On Windows (PowerShell), set it for the current session:
34
35 ```powershell
36 $env:HASDATA_API_KEY = "YOUR_KEY"
37 ```
38
39 Or persist it for future sessions:
40
41 ```powershell
42 [Environment]::SetEnvironmentVariable("HASDATA_API_KEY", "YOUR_KEY", "User")
43 ```
44
45- HTTP client with a configurable timeout of **at least 300 seconds** (see Pitfalls).
46- Network egress to `https://api.hasdata.com`.
47
48## Three Execution Modes
49
50| Mode | Latency | When | Endpoint |
51|---|---|---|---|
52| **Web Scraping API** | seconds | Arbitrary URL — JS rendering, CSS/AI extraction, screenshots | `POST /scrape/web` |
53| **Scraper APIs** (sync) | seconds | Pre-parsed JSON for known platforms (Google, Amazon, Zillow, …) | `GET /scrape/<vertical>/<resource>` |
54| **Scraper Jobs** (async) | minutes–hours | Bulk extraction, recursive crawling, webhook fan-out | `POST /scrapers/<slug>/jobs` |
55
56**Decision rule.** Default to a **Scraper API** when one exists for the target platform (pre-parsed JSON, no selector maintenance). Use **Web Scraping** for arbitrary URLs not covered by an API. Reach for a **Scraper Job** only when no API equivalent exists — `crawler`, `contacts`, `sec-edgar`, `amazon-bestsellers`, `amazon-product-reviews` — *or* when async fan-out + webhooks save engineering time over a paginated client loop.
57
58## Always-True Response Shape
59
60```json
61{
62 "requestMetadata": { "id": "…", "status": "ok", "url": "…" },
63 "...": "endpoint-specific"
64}
65```
66
67**HARD RULE:** Treat data as valid only if `requestMetadata.status === "ok"`. HTTP 200 alone is **not** a success signal.
68
69## Procedure
70
71### 1. Quick smoke test (Google SERP)
72
73```powershell
74curl -G 'https://api.hasdata.com/scrape/google/serp' `
75 --data-urlencode 'q=coffee' `
76 -H 'x-api-key: YOUR_KEY'
77```
78
79Expected: JSON with `requestMetadata.status` equal to `"ok"`.
80
81### 2. Choose the execution mode
82
83- **Scraper API** — if the platform has a dedicated endpoint (Google, Amazon, Zillow, Airbnb, Indeed, YouTube, etc.). Load the matching reference file (see Reference Loading Guide below) for parameters.
84- **Web Scraping API** — for arbitrary URLs. Load `references/web-scraping.md`.
85- **Scraper Job** — for bulk/async. Load `references/scraper-jobs.md`.
86
87### 3. Web Scraping API (`POST /scrape/web`)
88
89Load `references/web-scraping.md` before constructing the request.
90
91```powershell
92curl -X POST 'https://api.hasdata.com/scrape/web' `
93 -H 'x-api-key: YOUR_KEY' `
94 -H 'Content-Type: application/json' `
95 -d '{\"url\":\"https://example.com\",\"jsRendering\":false,\"outputFormat\":\"markdown\"}'
96```
97
98Key parameters (full list in reference):
99
100- `url` — target URL (required).
101- `jsRendering` — **default to `false`**. Enable only if the page requires a headless browser.
102- `outputFormat` — `markdown`, `html`, `text`, or `screenshot`.
103- `extract` — CSS selectors or AI extraction prompt.
104- `headers` — pass custom headers including `Cookie` (there is **no** top-level `cookies` parameter).
105
106### 4. Scraper APIs (sync, `GET /scrape/<vertical>/<resource>`)
107
108Load the relevant reference file for the vertical:
109
110```powershell
111curl -G 'https://api.hasdata.com/scrape/google/serp' `
112 --data-urlencode 'q=coffee' `
113 --data-urlencode 'num=10' `
114 -H 'x-api-key: YOUR_KEY'
115```
116
117Check `requestMetadata.status === "ok"` before consuming results. Inspect rich-snippet fields (`knowledgeGraph`, `localResults`, `inlineShoppingResults`, `relatedQuestions`) before considering direct page access.
118
119### 5. Scraper Jobs (async, `POST /scrapers/<slug>/jobs`)
120
121Load `references/scraper-jobs.md` before constructing the request.
122
123```powershell
124curl -X POST 'https://api.hasdata.com/scrapers/crawler/jobs' `
125 -H 'x-api-key: YOUR_KEY' `
126 -H 'Content-Type: application/json' `
127 -d '{\"url\":\"https://example.com\",\"outputFormat\":[\"markdown\"],\"includePaths\":\"/docs/.+\"}'
128```
129
130**HARD RULES for async jobs:**
131
1321. The submit response handle is `body.id` (an **integer**), **not** `jobId`. Persist it immediately.
1332. Poll `GET /scrapers/jobs/<id>` every 10–30 s with backoff.
1343. Treat webhooks as **best-effort** (3 retries). Always pair with polling.
1354. On `finished`, the status carries `data: {csv, json, xlsx}` short-lived URLs — **download immediately**.
136
137### 6. High-leverage patterns
138
139- **SERP-first enrichment.** Use Google SERP to surface public snippets for company/professional lookup before attempting direct scraping. Treat personal email/phone lookup as allowed only with a legitimate purpose and user authorization.
140- **AI Mode + verify.** `/scrape/google/ai-mode` for the answer + references → `/scrape/web` (markdown) on each reference URL → cited RAG context, no vector DB.
141- **Maps → leads.** `/scrape/google-maps/search` returns business websites and phones. Apply opt-out, rate, and privacy-law constraints before any outreach use.
142- **Crawler → corpus.** `crawler` Scraper Job with `outputFormat: ["markdown"]` + `includePaths: "/docs/.+"` produces an LLM-ready corpus in one submission.
143- **Pre-extracted via SERP rich snippets.** `knowledgeGraph`, `localResults`, `inlineShoppingResults`, `relatedQuestions` carry pre-parsed public facts. Always check them before considering direct page access.
144
145## Reference Loading Guide
146
147Load these files **on demand** — when the user's request maps to a specific vertical or pattern:
148
149| Reference file | Load when… |
150|---|---|
151| `references/web-scraping.md` | Using `POST /scrape/web` — parameters, JS scenarios, AI extraction, cookie auth. |
152| `references/search.md` | Google SERP / Light / AI Mode / News / Shopping / Bing / Trends + pagination. |
153| `references/ecommerce.md` | Amazon (product, search, seller, seller-products) and Shopify. |
154| `references/real-estate.md` | Zillow, Redfin (bracketed filters). |
155| `references/travel.md` | Airbnb, Booking, Google Flights (occupancy rules, token pagination, IATA codes). |
156| `references/local-business.md` | Maps (search/place/reviews/photos/posts), Yelp, YellowPages. |
157| `references/jobs.md` | Indeed and Glassdoor. |
158| `references/youtube.md` | YouTube search / video / channel / transcript. |
159| `references/scraper-jobs.md` | Async submit/poll/results, Crawler, Contacts, SEC EDGAR, webhook receiver. |
160| `references/code-recipes.md` | Ready-to-paste Python and TypeScript clients with retry, backoff, bounded concurrency, and the full job lifecycle. |
161
162## Wiring from Code
163
164- **Auth:** `x-api-key` header on every request. Read from `HASDATA_API_KEY` env. Never hardcode, never log.
165- **Timeouts:** Set client timeout **≥ 300 s**. HasData's own deadline is 300 s; shorter clients produce phantom failures while still being billed on completion.
166- **Retries:** `429` and `5xx` only — exponential backoff with jitter. **Never retry `4xx`** (auth, validation).
167- **Concurrency:** Cap at your plan limit. The free tier is **1**; anything higher just generates `429`s.
168- **Async jobs:** Handle is `body.id` (integer), not `jobId`. Poll `GET /scrapers/jobs/<id>` every 10–30 s. Webhooks are best-effort; always pair with polling. Download result URLs immediately on `finished`.
169
170See `references/code-recipes.md` for complete Python and TypeScript client implementations.
171
172## Pitfalls
173
174- **300 s server deadline.** Match your client timeout to at least 300 s. Shorter timeouts cause phantom failures while still being billed.
175- **`requestMetadata.status === "ok"` is the only success signal.** HTTP 200 alone is not enough — always check the metadata status field.
176- **Disable `jsRendering` first.** Enable only if the page needs it — most static pages parse fine without a headless browser. Rendering costs more and is slower.
177- **No `cookies` parameter.** Cookies go through `headers["Cookie"]`, not a top-level field.
178- **`includePaths` regex is case-sensitive.** `/blog/.+` will not match `/Blog/...`.
179- **Scraper Job `data` is double-wrapped.** Each row is `body.data[i].data`; the outer wrapper carries `id`, `jobId`, `dataId`, `createdAt`, `updatedAt`.
180- **Async job handle is `body.id` (integer), not `jobId`.** Persist it immediately after submit.
181- **Webhooks are best-effort with 3 retries.** Always have a polling fallback.
182- **Concurrency cap.** Free tier is 1 concurrent request. Exceeding your plan limit generates `429`s.
183- **Never retry `4xx`.** `401` = invalid key, `403` = quota exhausted, `429` = concurrency cap. Only `429` and `5xx` are retryable.
184- **Result URLs are short-lived.** Download `csv`/`json`/`xlsx` URLs immediately on job `finished`.
185
186## Verification
187
1881. **API key validity** — run the smoke test and confirm `requestMetadata.status === "ok"`:
189
190 ```powershell
191 curl -G 'https://api.hasdata.com/scrape/google/serp' `
192 --data-urlencode 'q=test' `
193 -H "x-api-key: $env:HASDATA_API_KEY"
194 ```
195
196 Expected output contains: `"status": "ok"`.
197
1982. **Error code mapping** — verify you handle each correctly:
199
200 | HTTP | Meaning | Retry? |
201 |---|---|---|
202 | `401` | Invalid key | No |
203 | `403` | Quota exhausted | No |
204 | `429` | Concurrency cap | Yes (backoff) |
205 | `500` | Server error | Yes (backoff) |
206
2073. **Async job lifecycle** — after `POST /scrapers/<slug>/jobs`, verify:
208 - Response contains `id` (integer) — not `jobId`.
209 - `GET /scrapers/jobs/<id>` eventually returns status `finished`.
210 - `data.csv`, `data.json`, or `data.xlsx` URLs are present and downloadable.
211
2124. **Client timeout** — confirm your HTTP client timeout is set to **≥ 300 s**.
213
214## Resources
215
216- Sitemap: <https://docs.hasdata.com/llms.txt>
217- API status codes: <https://docs.hasdata.com/api-codes>
218- Credits & concurrency: <https://docs.hasdata.com/credits-and-concurrency>
219- Dashboard: <https://app.hasdata.com>
220
221## Limitations
222
223- Requires access to HasData services and valid credentials.
224- Data quality and available fields depend on the target website and extraction method used.
225- JavaScript-heavy websites may require rendering, which can affect performance and cost.
226- Use only for public data or content the user is authorized to access; respect site terms, robots/access controls, privacy law, and rate limits.
227- Rate limits, quotas, and account restrictions may apply depending on the endpoint and subscription plan.