Web Scraping
What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Bulk fetching from hosts you do not own: how fast you may go, when a paid transport beats a direct
fetch, and how to prove the fallback you are relying on has ever run.
Contents
Iron Law: the rate is computed, not chosen
workers / sleep_seconds is the policy. A worker count with no arithmetic beside it is not a tuning
decision — it is an unreviewed multiplication of load on somebody else's server, usually from an IP
the institution owns and cannot swap. Raising 1 worker to 8 to "help the run finish faster" is not
helpful if it is the run that gets the university's address throttled: the user then loses the whole
corpus, not the hours you saved.
# ceiling: SEC EDGAR publishes 10 req/s and requires a declared User-Agent
# https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data (read 2026-09-08)
WORKERS, SLEEP_SECONDS = 8, 1.0
EFFECTIVE_RPS = WORKERS / SLEEP_SECONDS # 8.0 req/s = 80% of the documented ceiling
assert EFFECTIVE_RPS <= 10.0
No published ceiling → say so with the URL and the date you looked, and hold at 1 req/s.
"Not documented" is not "unlimited".
Quota vs rate limit
Name which one the target enforces, in a comment next to the worker count. The two have opposite
corrections, so a client that has not named one has guessed.
|
caps |
more workers → |
| Rate limit |
requests per unit time |
hits the cap sooner per second; total budget untouched, throttled requests retry, job still completes |
| Quota |
total requests per IP/key per period |
does not reduce total requests — only exhausts the budget sooner |
Under a quota, parallelising buys nothing on the binding constraint and brings the wall forward.
Under a rate limit, parallelising up to the ceiling is exactly right.
A fact recorded in a module docstring but never connected to the worker count is a fact nobody
applied. Put it where the decision is.
Verify the fallback before you rely on it
A declared fallback transport — proxy, unblocker, mirror, secondary API — is unverified until it
has executed. Count it; do not read about it.
xan frequency -s transport data/output/manifest.csv
# direct,1538
# proxy,0 <- documented as required; has never once run
Zero uses forces one of three, stated explicitly:
- Exercise it — force the fallback on a 5–10 row test, record the result, keep the dependency.
- Downgrade the claim — mark it
UNVERIFIED — 0 executions as of <date> everywhere it appears.
- Delete it — dependency, credentials and docs together.
Leaving a never-run path documented as required is the worst option: it makes an untested branch
look like a safety margin, so the run proceeds at a pressure the fallback was supposed to justify.
Direct fetch or paid transport
| Situation |
Use |
| Public data, documented ceiling, no anti-bot |
Direct fetch at the computed rate. A proxy here buys nothing and bills per request. |
| Per-IP quota you will exhaust |
Paid transport — it is the only thing that changes the binding constraint, because it changes the IP. More local workers cannot. |
| Anti-bot / CAPTCHA / fingerprinting |
Paid unblocker. Rolling your own is a maintenance treadmill. |
| Geo-restricted content |
Proxy with geo-targeting. |
| Data already collected by a vendor |
Buy the dataset or archive dump; do not re-crawl. |
Vendor table
Every figure below was read from the vendor's own page on 2026-09-08; URLs in
References. Pricing and limits move — re-read before quoting them to anyone.
|
Zyte API |
Bright Data |
ScrapingBee |
| Throughput model |
Rate limit, RPM-based — 3,000 RPM standard, 10,000 RPM enterprise, per API key. Not a concurrency cap. |
Per-zone concurrency + spend caps. Vendor claims no concurrency limit (SERP pricing FAQ); no canonical docs page states a number — treat as unverified. |
Concurrency cap, per plan — 25 / 50 / 100 / 200 / 400 concurrent requests, Hobby → Business+. |
| Over-limit behaviour |
HTTP 429, charged nothing, retry with backoff; official clients retry by default. |
Per-zone auto-throttling on the target; spend limit is the real guardrail. |
429/5xx from the target; failed attempts still burn credits unless mode=auto fails entirely (0 credits). |
| Client concurrency knob |
AsyncZyteAPI(n_conn=...), CONCURRENT_REQUESTS in Scrapy; docs use ~15 as the worked example. |
Zone config; raise via support. |
Whatever your plan's cap is — cap per domain, not just globally. |
| Billing unit |
Per successful response; tier depends on target + HTTP vs browser. Screenshots $0.002; auto-extraction $0.0004–$0.0016/type. |
Per successful request: Unlocker/SERP/Crawl from $1/1k req; Browser API from $5/GB; Web Archive dump ~$0.001/page; datasets from $250/100k rec. |
Credits: 1 plain, 5 JS render, 10 premium proxy no-JS, 25 premium+JS, 75 stealth. Plans $19–$599/mo for 75k–8M credits. |
| Entry price |
PAYG on signup, $5 free credit, $100/mo spend limit. |
Free tier on Unlocker/SERP/Scraper APIs. |
1,000 free credits, no card. |
| Best at |
Scrapy-native pipelines; explicit published rate limits you can compute against. |
Breadth — archive corpus, prebuilt datasets, unlocker, all under one token. Already wired for FINRA/IAPD work. |
Simple REST unblocking with a hard, legible concurrency number per plan. |
Do not read the concurrency column as a target. It is the vendor's ceiling on their side; the
target site's ceiling is a different number and is the one E7 measures against. ScrapingBee's own
docs warn that running at the cap provokes 429/5xx from targets and burns credits on failures.
Facts
- SEC EDGAR publishes 10 requests/second and requires a declared
User-Agent naming a contact
address; it explicitly reserves the right to throttle and states it does not allow botnets or
automated crawling outside that policy. A scraper that neither declares a UA nor computes its rate
is not merely impolite — it is the case the policy names, and the block lands on the shared IP.
- Rate-limited Zyte requests cost nothing, so retrying with backoff is free; a hand-rolled
client that treats 429 as a fatal error throws away a retry the vendor already priced at zero.
- ScrapingBee credits are not requests. JS rendering is 5×, premium proxy with JS is 25×, stealth
is 75×. A 1M-credit plan is 1M pages only if nothing renders JS — sizing a job in "requests"
against a credit allowance overstates capacity by up to 75×.
- Bright Data's "unlimited concurrency" appears in pricing/marketing copy, not in a docs page with
a number. Quoting it as a verified limit is an unverified claim presented as fact; if a run
depends on it, open a ticket and get the zone provisioned rather than assuming.
- A per-IP quota is the one case where local concurrency cannot help at all. Parallelising into a
quota is the shape of work that looks like a 3× speedup in the log and is a 0× improvement on what
the run can actually retrieve.
Red flags — STOP
| About to |
Why wrong |
Do instead |
Set max_workers=N with no EFFECTIVE_RPS beside it |
The load change is unreviewed and unrecorded |
Compute workers / sleep, name the ceiling and its URL |
| Raise workers because the job is slow |
If the target enforces a quota this buys nothing and exhausts it sooner |
Name quota vs rate limit first; that decides the fix |
| Treat "no documented rate limit" as permission |
Absence of a published number is not consent |
Hold at 1 req/s and record where you looked |
| Cite a fallback proxy as the answer to a quota |
It may never have executed |
xan frequency -s transport on the manifest; zero uses = untested |
| Quote a vendor concurrency or price figure from memory |
These pages changed within the last year |
Re-read the vendor page and date the citation |
| Size a ScrapingBee job in requests against a credit allowance |
Credits are 1–75× per request |
Multiply by the tier you will actually use |
| Call a paid unblocker for public, unprotected pages |
Bills per request for something a direct fetch does free |
Direct fetch at the computed rate |
| Run a bulk scrape without a 5–10 page test first |
Repo rule: test before scaling |
Fetch 5–10, inspect, then scale |
References
Read on 2026-09-08:
Local:
${CLAUDE_SKILL_DIR}/../../rules/ds-rate-limit-choice.md — constraint E7, normative.
${CLAUDE_SKILL_DIR}/references/bright-data-api.md — Bright Data API, auth, cost enforcement, endpoints.
${CLAUDE_SKILL_DIR}/references/bright-data-webarchive-api.md — Web Archive filters and parallel-poll harness.
${CLAUDE_SKILL_DIR}/references/bright-data-datasets-catalog.md — the 1,576-dataset marketplace.
${CLAUDE_SKILL_DIR}/references/bright-data-finra-sec-coverage.md — verified FINRA/IAPD coverage and dump costs.
- The
sec-fetch skill owns SEC.gov specifically (declared-UA access that avoids the 403).
1---2name: web-scraping3version: 1.04description: Use when "scrape this site at scale", "fetch N thousand pages", "how many workers should this scraper use", "am I hammering this server", "we're getting 429s / blocked / CAPTCHAs", "should we use a proxy", "Zyte", "ScrapingBee", "Web Unlocker", "residential proxy", "rotate proxies", "scrape with Web Unlocker", "query Bright Data", "Bright Data datasets", "Bright Data Web Archive / Wayback alternative", "FINRA BrokerCheck data", "SEC IAPD / adviserinfo data", "Investment Adviser Public Disclosure", "broker/adviser disclosure snapshots", "LinkedIn/Crunchbase/Glassdoor company or people dataset", or any use of the Bright Data API (datasets/list, Web Archive search/dump, Web Unlocker zones).5user-invocable: false6metadata:7 source: https://skillmd.com/skills/edwinhu/web-scraping8---910<!-- wc-probe: ignore-narrated-history -->11<!-- vendor pricing/limits vintage (2026-09-08) is the date the figures were read -->1213# Web Scraping1415**What this skill carries** — grep `references/` for any subject the names below miss:16!`d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"`1718Bulk fetching from hosts you do not own: how fast you may go, when a paid transport beats a direct19fetch, and how to prove the fallback you are relying on has ever run.2021## Contents2223- [Iron Law: the rate is computed, not chosen](#iron-law-the-rate-is-computed-not-chosen)24- [Quota vs rate limit](#quota-vs-rate-limit)25- [Verify the fallback before you rely on it](#verify-the-fallback-before-you-rely-on-it)26- [Direct fetch or paid transport](#direct-fetch-or-paid-transport)27- [Vendor table](#vendor-table)28- [Facts](#facts)29- [Red flags — STOP](#red-flags--stop)30- [References](#references)3132## Iron Law: the rate is computed, not chosen3334<EXTREMELY-IMPORTANT>35**NO CONCURRENT NETWORK CLIENT WITHOUT A COMPUTED EFFECTIVE REQUEST RATE AND THE TARGET'S36DOCUMENTED CEILING, BOTH WRITTEN IN THE CODE.** This is constraint **E7** in the ds engineering37set — `${CLAUDE_SKILL_DIR}/../../rules/ds-rate-limit-choice.md` is the normative38text; this skill is its vendor-facing half.3940`workers / sleep_seconds` is the policy. A worker count with no arithmetic beside it is not a tuning41decision — it is an unreviewed multiplication of load on somebody else's server, usually from an IP42the institution owns and cannot swap. Raising 1 worker to 8 to "help the run finish faster" is not43helpful if it is the run that gets the university's address throttled: the user then loses the whole44corpus, not the hours you saved.45</EXTREMELY-IMPORTANT>4647```python48# ceiling: SEC EDGAR publishes 10 req/s and requires a declared User-Agent49# https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data (read 2026-09-08)50WORKERS, SLEEP_SECONDS = 8, 1.051EFFECTIVE_RPS = WORKERS / SLEEP_SECONDS # 8.0 req/s = 80% of the documented ceiling52assert EFFECTIVE_RPS <= 10.053```5455No published ceiling → say so with the URL and the date you looked, and hold at 1 req/s.56"Not documented" is not "unlimited".5758## Quota vs rate limit5960Name which one the target enforces, in a comment next to the worker count. The two have **opposite**61corrections, so a client that has not named one has guessed.6263| | caps | more workers → |64|---|---|---|65| **Rate limit** | requests per unit time | hits the cap sooner *per second*; total budget untouched, throttled requests retry, job still completes |66| **Quota** | total requests per IP/key per period | does **not** reduce total requests — only **exhausts the budget sooner** |6768Under a quota, parallelising buys nothing on the binding constraint and brings the wall forward.69Under a rate limit, parallelising up to the ceiling is exactly right.7071A fact recorded in a module docstring but never connected to the worker count is a fact nobody72applied. Put it where the decision is.7374## Verify the fallback before you rely on it7576A declared fallback transport — proxy, unblocker, mirror, secondary API — is **unverified until it77has executed**. Count it; do not read about it.7879```bash80xan frequency -s transport data/output/manifest.csv81# direct,153882# proxy,0 <- documented as required; has never once run83```8485Zero uses forces one of three, stated explicitly:86871. **Exercise it** — force the fallback on a 5–10 row test, record the result, keep the dependency.882. **Downgrade the claim** — mark it `UNVERIFIED — 0 executions as of <date>` everywhere it appears.893. **Delete it** — dependency, credentials and docs together.9091Leaving a never-run path documented as required is the worst option: it makes an untested branch92look like a safety margin, so the run proceeds at a pressure the fallback was supposed to justify.9394## Direct fetch or paid transport9596| Situation | Use |97|---|---|98| Public data, documented ceiling, no anti-bot | **Direct fetch** at the computed rate. A proxy here buys nothing and bills per request. |99| Per-IP **quota** you will exhaust | Paid transport — it is the only thing that changes the binding constraint, because it changes the IP. More local workers cannot. |100| Anti-bot / CAPTCHA / fingerprinting | Paid unblocker. Rolling your own is a maintenance treadmill. |101| Geo-restricted content | Proxy with geo-targeting. |102| Data already collected by a vendor | Buy the dataset or archive dump; do not re-crawl. |103104## Vendor table105106Every figure below was read from the vendor's own page on **2026-09-08**; URLs in107[References](#references). Pricing and limits move — re-read before quoting them to anyone.108109| | Zyte API | Bright Data | ScrapingBee |110|---|---|---|---|111| **Throughput model** | **Rate limit, RPM-based** — 3,000 RPM standard, 10,000 RPM enterprise, per API key. Not a concurrency cap. | Per-zone concurrency + spend caps. **Vendor claims no concurrency limit** (SERP pricing FAQ); no canonical docs page states a number — treat as unverified. | **Concurrency cap, per plan** — 25 / 50 / 100 / 200 / 400 concurrent requests, Hobby → Business+. |112| **Over-limit behaviour** | HTTP 429, **charged nothing**, retry with backoff; official clients retry by default. | Per-zone auto-throttling on the target; spend limit is the real guardrail. | 429/5xx from the target; failed attempts still burn credits unless `mode=auto` fails entirely (0 credits). |113| **Client concurrency knob** | `AsyncZyteAPI(n_conn=...)`, `CONCURRENT_REQUESTS` in Scrapy; docs use ~15 as the worked example. | Zone config; raise via support. | Whatever your plan's cap is — cap **per domain**, not just globally. |114| **Billing unit** | Per **successful** response; tier depends on target + HTTP vs browser. Screenshots $0.002; auto-extraction $0.0004–$0.0016/type. | Per successful request: Unlocker/SERP/Crawl from **$1/1k req**; Browser API from $5/GB; Web Archive dump ~$0.001/page; datasets from $250/100k rec. | **Credits**: 1 plain, 5 JS render, 10 premium proxy no-JS, 25 premium+JS, 75 stealth. Plans $19–$599/mo for 75k–8M credits. |115| **Entry price** | PAYG on signup, $5 free credit, $100/mo spend limit. | Free tier on Unlocker/SERP/Scraper APIs. | 1,000 free credits, no card. |116| **Best at** | Scrapy-native pipelines; explicit published rate limits you can compute against. | Breadth — archive corpus, prebuilt datasets, unlocker, all under one token. Already wired for FINRA/IAPD work. | Simple REST unblocking with a hard, legible concurrency number per plan. |117118**Do not read the concurrency column as a target.** It is the vendor's ceiling on *their* side; the119target site's ceiling is a different number and is the one E7 measures against. ScrapingBee's own120docs warn that running at the cap provokes 429/5xx from targets and burns credits on failures.121122## Facts123124- SEC EDGAR publishes **10 requests/second** and requires a declared `User-Agent` naming a contact125 address; it explicitly reserves the right to throttle and states it does not allow botnets or126 automated crawling outside that policy. A scraper that neither declares a UA nor computes its rate127 is not merely impolite — it is the case the policy names, and the block lands on the shared IP.128- **Rate-limited Zyte requests cost nothing**, so retrying with backoff is free; a hand-rolled129 client that treats 429 as a fatal error throws away a retry the vendor already priced at zero.130- **ScrapingBee credits are not requests.** JS rendering is 5×, premium proxy with JS is 25×, stealth131 is 75×. A 1M-credit plan is 1M pages only if nothing renders JS — sizing a job in "requests"132 against a credit allowance overstates capacity by up to 75×.133- **Bright Data's "unlimited concurrency" appears in pricing/marketing copy, not in a docs page with134 a number.** Quoting it as a verified limit is an unverified claim presented as fact; if a run135 depends on it, open a ticket and get the zone provisioned rather than assuming.136- A per-IP **quota** is the one case where local concurrency cannot help at all. Parallelising into a137 quota is the shape of work that looks like a 3× speedup in the log and is a 0× improvement on what138 the run can actually retrieve.139140## Red flags — STOP141142| About to | Why wrong | Do instead |143|---|---|---|144| Set `max_workers=N` with no `EFFECTIVE_RPS` beside it | The load change is unreviewed and unrecorded | Compute `workers / sleep`, name the ceiling and its URL |145| Raise workers because the job is slow | If the target enforces a quota this buys nothing and exhausts it sooner | Name quota vs rate limit first; that decides the fix |146| Treat "no documented rate limit" as permission | Absence of a published number is not consent | Hold at 1 req/s and record where you looked |147| Cite a fallback proxy as the answer to a quota | It may never have executed | `xan frequency -s transport` on the manifest; zero uses = untested |148| Quote a vendor concurrency or price figure from memory | These pages changed within the last year | Re-read the vendor page and date the citation |149| Size a ScrapingBee job in requests against a credit allowance | Credits are 1–75× per request | Multiply by the tier you will actually use |150| Call a paid unblocker for public, unprotected pages | Bills per request for something a direct fetch does free | Direct fetch at the computed rate |151| Run a bulk scrape without a 5–10 page test first | Repo rule: test before scaling | Fetch 5–10, inspect, then scale |152153## References154155Read on **2026-09-08**:156157- Zyte rate limits — https://docs.zyte.com/zyte-api/usage/rate-limit.html158- Zyte pricing (3,000/10,000 RPM, spend limits, per-feature costs) — https://docs.zyte.com/zyte-api/pricing.html159- Zyte parallelism guidance — https://docs.zyte.com/zyte-api/usage/optimize.html160- ScrapingBee pricing (plans, credits, concurrency) — https://www.scrapingbee.com/pricing/161- ScrapingBee credit multipliers and `mode=auto` — https://www.scrapingbee.com/documentation/162- Bright Data Web Unlocker overview — https://docs.brightdata.com/scraping-automation/web-unlocker/introduction163- Bright Data pricing — https://brightdata.com/pricing/web-unlocker164- SEC EDGAR fair-access policy — https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data165166Local:167168- `${CLAUDE_SKILL_DIR}/../../rules/ds-rate-limit-choice.md` — constraint E7, normative.169- `${CLAUDE_SKILL_DIR}/references/bright-data-api.md` — Bright Data API, auth, cost enforcement, endpoints.170- `${CLAUDE_SKILL_DIR}/references/bright-data-webarchive-api.md` — Web Archive filters and parallel-poll harness.171- `${CLAUDE_SKILL_DIR}/references/bright-data-datasets-catalog.md` — the 1,576-dataset marketplace.172- `${CLAUDE_SKILL_DIR}/references/bright-data-finra-sec-coverage.md` — verified FINRA/IAPD coverage and dump costs.173- The `sec-fetch` skill owns SEC.gov specifically (declared-UA access that avoids the 403).