Bright Data — Scrape
Get clean content (markdown, HTML, JSON, screenshot) from one or more URLs via the Bright Data CLI. This skill owns the "fetch raw or lightly-structured content" job. For platform-specific structured data (Amazon, LinkedIn, TikTok, etc.), stop and use data-feeds instead — you'll get clean JSON without selector logic.
Setup gate (run first)
Before any scrape, verify the CLI is installed and authenticated:
if ! command -v bdata >/dev/null 2>&1; then
echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md"
elif ! bdata zones >/dev/null 2>&1; then
echo "bdata not authenticated — run: bdata login (or: bdata login --device for SSH)"
fi
If either check fails, halt and route the user to skills/bright-data-best-practices/references/cli-setup.md. Do not attempt the legacy curl fallback silently — ask the user first.
Pick your path
| Situation |
Action |
| Single URL |
bdata scrape <url> -f markdown |
| Small list (≤ ~20 URLs) |
shell loop, 1 at a time (see references/patterns.md) |
| Larger list (dozens+) |
xargs -P 4 with parallelism cap (see references/patterns.md) |
| Paginated listing |
scrape page 1 → extract next-page URL → append → repeat (see references/examples.md) |
| JS-heavy / login-gated / interaction-required |
escalate to bdata browser (see brightdata-cli skill) |
| Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, … |
stop — hand off to data-feeds |
| No URL yet, just a topic |
hand off to search |
Action
Core commands:
# Clean markdown (default)
bdata scrape "https://example.com/article" -f markdown -o article.md
# Raw HTML (when you need the DOM)
bdata scrape "https://example.com" -f html -o page.html
# Structured JSON (when the Unlocker returns parsed fields)
bdata scrape "https://example.com" -f json --pretty -o page.json
# Visual snapshot (saves PNG)
bdata scrape "https://example.com" -f screenshot -o page.png
# Geo-targeted (override the exit country)
bdata scrape "https://example.com" --country de -f markdown
Full flag reference: references/flags.md.
Verification gate (run before claiming success)
- Non-empty output:
test -s "$out_path" — or, for stdout, at least 200 bytes of content.
- Not a block page — grep the output for any of these signatures (case-insensitive):
Access DeniedJust a momentAttention RequiredChecking your browsercaptchacf-browser-verificationcloudflare (with < 2KB total body)
- Expected markers present for the task: e.g., a product page should contain a price pattern (
\$\d); an article should contain at least one <h1> or # heading.
- On failure, escalation ladder:
- Retry with a different
--country (e.g., --country de if the origin site is US)
- Escalate to
bdata browser for full JS rendering (hand off to brightdata-cli skill)
Do not report success until all checks above pass.
Red flags
- Claiming success without inspecting the output.
- Silencing errors with
2>/dev/null — you'll miss auth failures and rate-limit errors.
- Running
bdata scrape on Amazon/LinkedIn/TikTok/Instagram/YouTube/Reddit URLs — these are supported by data-feeds and return structured data directly. Scraping loses the structure.
- Scraping the same URL repeatedly in the same task — cache the first result.
- Looping
bdata scrape sequentially for large lists instead of using xargs -P 4 (or similar) with a parallelism cap.
- Using
curl against api.brightdata.com directly — legacy path; only when the CLI isn't available.
References
references/flags.md — every flag with when-to-use notes.references/patterns.md — shell-loop batching, xargs parallelism, pagination recipe, retry/backoff, block-page recovery chain, legacy curl fallback.references/examples.md — (1) single page → markdown, (2) batch a list of URLs with parallelism cap, (3) paginated listing, (4) block-page recovery.
1---2name: scrape3description: Scrape web content as clean markdown/HTML/JSON via the Bright Data CLI (`bdata scrape`). Use when the user wants to fetch a page, extract content from a list of URLs, or crawl paginated listings. Hands off to `data-feeds` for supported platforms (Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc.) and to `search` when URLs must be discovered first. Requires the Bright Data CLI; proactively guides install + login if missing.4---5
6# Bright Data — Scrape
7
8Get clean content (markdown, HTML, JSON, screenshot) from one or more URLs via the Bright Data CLI. This skill owns the "fetch raw or lightly-structured content" job. For platform-specific structured data (Amazon, LinkedIn, TikTok, etc.), **stop and use `data-feeds` instead** — you'll get clean JSON without selector logic.
9
10## Setup gate (run first)
11
12Before any scrape, verify the CLI is installed and authenticated:
13
14```bash
15if ! command -v bdata >/dev/null 2>&1; then
16 echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md"
17elif ! bdata zones >/dev/null 2>&1; then
18 echo "bdata not authenticated — run: bdata login (or: bdata login --device for SSH)"
19fi
20```
21
22If either check fails, halt and route the user to `skills/bright-data-best-practices/references/cli-setup.md`. Do not attempt the legacy `curl` fallback silently — ask the user first.
23
24## Pick your path
25
26| Situation | Action |
27|---|---|
28| Single URL | `bdata scrape <url> -f markdown` |
29| Small list (≤ ~20 URLs) | shell loop, 1 at a time (see `references/patterns.md`) |
30| Larger list (dozens+) | `xargs -P 4` with parallelism cap (see `references/patterns.md`) |
31| Paginated listing | scrape page 1 → extract next-page URL → append → repeat (see `references/examples.md`) |
32| JS-heavy / login-gated / interaction-required | escalate to `bdata browser` (see `brightdata-cli` skill) |
33| Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, … | **stop — hand off to `data-feeds`** |
34| No URL yet, just a topic | **hand off to `search`** |
35
36## Action
37
38Core commands:
39
40```bash
41# Clean markdown (default)
42bdata scrape "https://example.com/article" -f markdown -o article.md
43
44# Raw HTML (when you need the DOM)
45bdata scrape "https://example.com" -f html -o page.html
46
47# Structured JSON (when the Unlocker returns parsed fields)
48bdata scrape "https://example.com" -f json --pretty -o page.json
49
50# Visual snapshot (saves PNG)
51bdata scrape "https://example.com" -f screenshot -o page.png
52
53# Geo-targeted (override the exit country)
54bdata scrape "https://example.com" --country de -f markdown
55
56```
57
58Full flag reference: [`references/flags.md`](references/flags.md).
59
60## Verification gate (run before claiming success)
61
621. **Non-empty output:** `test -s "$out_path"` — or, for stdout, at least 200 bytes of content.
632. **Not a block page** — grep the output for any of these signatures (case-insensitive):
64 - `Access Denied`
65 - `Just a moment`
66 - `Attention Required`
67 - `Checking your browser`
68 - `captcha`
69 - `cf-browser-verification`
70 - `cloudflare` *(with < 2KB total body)*
713. **Expected markers present** for the task: e.g., a product page should contain a price pattern (`\$\d`); an article should contain at least one `<h1>` or `# ` heading.
724. **On failure, escalation ladder:**
73 - Retry with a different `--country` (e.g., `--country de` if the origin site is US)
74 - Escalate to `bdata browser` for full JS rendering (hand off to `brightdata-cli` skill)
75
76Do not report success until all checks above pass.
77
78## Red flags
79
80- Claiming success without inspecting the output.
81- Silencing errors with `2>/dev/null` — you'll miss auth failures and rate-limit errors.
82- Running `bdata scrape` on Amazon/LinkedIn/TikTok/Instagram/YouTube/Reddit URLs — these are supported by `data-feeds` and return structured data directly. Scraping loses the structure.
83- Scraping the same URL repeatedly in the same task — cache the first result.
84- Looping `bdata scrape` sequentially for large lists instead of using `xargs -P 4` (or similar) with a parallelism cap.
85- Using `curl` against `api.brightdata.com` directly — legacy path; only when the CLI isn't available.
86
87## References
88
89- [`references/flags.md`](references/flags.md) — every flag with when-to-use notes.
90- [`references/patterns.md`](references/patterns.md) — shell-loop batching, `xargs` parallelism, pagination recipe, retry/backoff, block-page recovery chain, legacy `curl` fallback.
91- [`references/examples.md`](references/examples.md) — (1) single page → markdown, (2) batch a list of URLs with parallelism cap, (3) paginated listing, (4) block-page recovery.
om-scogo