# Web Crawler

> Web scraping plus social data: YouTube, TikTok, Instagram (incl. stories, followers, likers, hashtags), LinkedIn, Reddit, Threads, plus China apps and Asian e-commerce via Apify (Douyin, Xiaohongshu, Weibo, Bilibili, Zhihu, Douban, Xueqiu, Taobao, JD, 1688, Xianyu, Shopee, Lazada, Temu, Trip.com reviews), plus robust web-page fallback extraction. Use when extracting public posts, transcripts, or pages JS-heavy enough to block plain fetch (e.g. download YouTube transcript, scrape TikTok comments, listing pages behind anti-bot/Cloudflare). Auto-fallback trigger terms: web_fetch failed, 403, anti-bot, cloudflare, JS-heavy page. Also use for whole-site link audits and "this button/link does not work" reports (check all links, broken links, 404, button not working) — you investigate with site-agnostic fetch / browse tools.

- Skill: `starchild-ai-agent/web-crawler` (Agent Skill, multi-file: 10 files)
- Install (CLI): `npx get-skill add starchild-ai-agent/web-crawler`
- Raw SKILL.md: https://api.skillmd.com/api/skills/starchild-ai-agent/web-crawler/raw
- Safety review: pending (external: skill-scanner PASS, skillspector CAUTION)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: starchild-ai-agent (https://skillmd.com/u/starchild-ai-agent)
- Updated: 2026-11-11
- Page: https://skillmd.com/skills/starchild-ai-agent/web-crawler

---


## Preferred entry: call exports.py (don't hand-roll requests)
Ready-made helpers live in `skills/web-crawler/exports.py`. Prefer them over
writing your own `proxied_get`/`proxied_post` calls — they already inject the
proxy credentials, so **there is no API key to find** (don't read
`$SCRAPECREATORS_API_KEY` / `$FIRECRAWL_API_KEY`, don't check `.env`, don't ask
the user).

```python
import sys; sys.path.insert(0, "/data/workspace/skills/web-crawler")
from exports import scrape_markdown, youtube_transcript, sc_get
scrape_markdown("https://example.com/article")          # Firecrawl fallback
youtube_transcript("https://youtube.com/watch?v=ID")     # ScrapeCreators
youtube_video("https://youtube.com/watch?v=ID")          # metadata: title/description/uploader (+ transcript)
sc_get("/v1/tiktok/profile", handle="charlidamelio")     # any SC endpoint

from exports import archive_fallback                      # paywall / Firecrawl-403
archive_fallback("https://www.nytimes.com/.../article.html")  # archive snapshot
```

### Transcript route — ONE entry for any media link

```python
from core.skill_tools import web_crawler
r = web_crawler.get_transcript(url, caller_id="chat:<thread>")
# {found, kind, source, text, url, title, tried, note}
```

**Design rule: captions / subtitles / published transcripts are METADATA;
the media file is PAYLOAD.** Exhaust every metadata provider first; touch the
media only when the user explicitly agrees. Same rule claude-video, spoken.md
and yt-dlp's `--skip-download` workflow follow. Nothing here is site-specific:

| provider (cost order) | what it is | identity / validity check |
|---|---|---|
| `media_info(url)` | yt-dlp `extract_info(download=False)` — 1800+ sites, returns title, duration, upload date, channel, caption-track URLs. Zero media bytes. | track fetched must be 2xx with a real body |
| RSS `<podcast:transcript>` | Podcasting 2.0 standard, any podcast | feed item matched by duration ±5 % + date ±3 d; transcript URL must be 2xx, > 500 chars, not an error page |
| show's own YouTube upload | channel = the YouTube URL the show declares in its RSS (else search hit whose channel name **equals** the show name minus stopwords) | candidates = duration ±5 % AND date ±3 d; **identity = title similarity ≥ 0.6 OR show-notes overlap** (≥ 6 shared distinctive tokens, Jaccard ≥ 0.15). A candidate that fails identity is returned in `candidate` with `found=False` — never delivered as the episode |
| `scrape_markdown(url)` | any other page (publisher, Snipd, blog) — browser-rendered, survives WAF 403 | > 1000 chars |

`found=False` ⇒ report `kind`, `title`, `tried`, `note` (and `candidate` if
any) and **ask the user** before any download. YouTube is recognised by URL:
the captions API runs even when yt-dlp metadata fails, and a page scrape is
never reported as a transcript for a media URL. `yt-dlp --skip-download` / `-J` / `--list-subs` are
metadata calls and pass the bash gate; `yt-dlp <url>`, `curl … .mp3`, ffmpeg,
whisper are held for confirmation.

Title only, no link → `web_search("<show> <title> transcript")` then
`get_transcript(<page url>)`.

Why: answering 403s on transcript hosts by downloading audio and running
whisper is slow, expensive and often incomplete. `get_transcript` instead
follows RSS → the channel the feed declares → the matching upload → captions,
with no audio download.

## Site audit — "check all links" / "this button doesn't work"

You are the auditor. `exports.py` gives you small, site-agnostic tools that
**observe and return raw data**; they never decide what is broken. There is no
keyword list, no per-site config and no verdict function — every site is
different, so you look, form a hypothesis, test it, and judge from evidence.

| Tool | Returns |
|---|---|
| `fetch(url, method="GET")` | raw `status`, `final_url`, headers, body, `error` |
| `check_urls(urls, method="HEAD")` | `fetch` for many URLs in parallel, no bodies |
| `extract_refs(html, base_url)` | every URL-like reference + where it came from (`a[href]`, `link[rel=…]`, `inline-script`, `attr[data-…]`, `meta[refresh]`…) |
| `parse_sitemap(xml)` | `<loc>` entries of a sitemap / sitemap index |
| `browse(url, actions=[…], viewport=…, mobile=…)` | real Chromium: rendered links (incl. shadow DOM), visible controls, JS exceptions, console errors, ≥400 responses, failed requests, and for each action you asked for: URL before/after, navigations with status, new windows with **their own** status, downloads, DOM changed?, new links |

Method (details and worked checks: `reference/site-audit-method.md`;
full worked example: `reference/site-audit-case-study.md`):

1. **Map the site from several sources, then reconcile.** Home page HTML via
   `extract_refs`, `robots.txt` → sitemap(s) via `parse_sitemap`, rendered DOM
   via `browse`, and any URL the user named. Read the `inline-script` and
   `attr[...]` hits yourself — route tables and JS navigation live there. A page
   the user mentions that none of your sources found is a finding about your
   map, not about the user.
2. **Learn what "normal" and "error" look like on THIS site.** Fetch one real
   page and one surely-missing URL (random path). Compare status, title, size,
   text. That baseline — not a word list — tells you how this site shows a 404
   (real 404, 200 soft-404, redirect to home…).
3. **Check every page and every reference.** `check_urls`, re-try HEAD failures
   with GET, re-check transient errors once. Keep raw results; group them
   yourself (same-site pages / assets / third-party).
4. **For what users actually click, click it.** Use `browse(...).controls` and
   rendered links to choose targets, then `browse(url, actions=[{"click": sel}])`
   at desktop **and** mobile width. Judge from the action record: did the URL
   change, did a new window open and what was *its* status, did a download
   finish, did the DOM change, did a JS error fire, was the element covered
   (`element.topmost_at_center`)?
5. **Explain anything unexpected before reporting it.** 401/403/429 or bot
   walls = "could not verify", not broken and not fine. A click with no visible
   effect may be correct (already on that page) — say what you observed.
6. **Report with evidence.** Pages found and how; what is broken (URL, where it
   is referenced, status/observation, viewport); what could not be verified and
   why; what you did not cover. Never write "all links OK" for areas you did not
   check.

Safety: only click targets you chose for a reason. Don't click anything
`form_associated` or whose text/context implies submit, buy, delete, send or
subscribe unless the user explicitly asked. `browse` aborts non-GET requests
for the whole browser session by default — page load, actions and any new
windows (`blocked_writes`, each with its `phase`); keep it that way unless the user
authorizes writes. If Playwright is unavailable `browse` returns `{"error": …}` —
report clicks as not verified; never infer them from status codes.

## Quick trigger rules (read this first)
Use this skill immediately when any of these conditions is true:

- `web_fetch` returns HTTP **401/403/429/5xx**
- Response is an anti-bot challenge page (for example Cloudflare "Attention Required", "Just a moment", or challenge/captcha pages)
- The page is JS-heavy and the first fetch misses required detail fields (for example publish time, author, listing code, updated time, price breakdown)
- Search results do not contain the requested field and the value must be extracted from the target page itself

Fallback rule:
- If ordinary fetch is blocked or incomplete, switch to Firecrawl fallback in this skill before asking the user for screenshots/manual text.

## Error signatures -> action
| Signature | Action |
|---|---|
| `web_fetch` HTTP 401/403/429/5xx | Call Firecrawl `POST /v2/scrape` once with `formats:["markdown","links"]` + `onlyMainContent:true` |
| Cloudflare/challenge page text in body | Same Firecrawl call as above |
| Markdown still misses key fields | Retry once with `formats:["rawHtml"]` |
| **Firecrawl returns 403 / empty for a social media URL** | Check the intent-routing tables below for a **ScrapeCreators platform-specific endpoint** for this domain (e.g. `sc_get('/v1/instagram/post', url=...)`, `sc_get('/v2/tiktok/video', url=...)`). If one exists, use it — these have dedicated extraction that bypasses anti-bot. If no dedicated endpoint exists, fall through to `archive_fallback` or ask the user. |
| **Firecrawl itself returns 403 / empty** (hard paywall: NYT, WSJ, Economist, FT, Bloomberg) | Call `archive_fallback(url)` — recovers full text from a web archive snapshot |
| **Need structured data from a China app** (抖音/小红书/微博/B站/京东/淘宝/1688/闲鱼/得物 etc.) | Call `apify_run()` — Apify Store has purpose-built actors for these platforms that Firecrawl/ScrapeCreators don't cover |
| **Instagram story URL** (`instagram.com/stories/<user>/<id>`) | ScrapeCreators has no story endpoint and Firecrawl hits the login wall — go straight to `apify_run("data-slayer~instagram-stories-scraper", ...)`. See "Instagram stories" below |

### Paywall / Firecrawl-blocked fallback chain (use `archive_fallback`)
When Firecrawl can't get the page either (it returns 403, or markdown comes back
empty) the site is behind a hard paywall or aggressive WAF. Do NOT keep retrying
Firecrawl. Recover the article from a **web archive snapshot** instead:

```python
from exports import archive_fallback
res = archive_fallback("https://www.nytimes.com/.../article.html")
res["markdown"]      # full text, or "" if no snapshot exists anywhere
res["source"]        # "archive.today" | "wayback" | None
res["snapshot_url"]  # the snapshot that was scraped
```

How it works (and why this order):
- **archive.today first** (`archive.ph` / `archive.is` mirrors). User-triggered,
  real-browser captures; historically preserves full text *behind* paywalls.
  Best bet for NYT/WSJ/Economist. We scrape its `/newest/` snapshot via Firecrawl
  (archive.today has its own Cloudflare, so scrape it through Firecrawl, never
  `web_fetch` it directly).
- **Wayback Machine second** (`archive.org`). Automated crawler that *honors*
  robots.txt and paywalls, so it often has NO full text for hard paywalls — but
  it's a good fallback for ordinary 403/Cloudflare pages that aren't paywalled.

Limitation: archives only return text **someone already saved**. If
`res["markdown"] == ""`, no snapshot exists — stop, tell the user, and try the
outlet's official API/RSS or a different source. Do not fabricate the article.

### China-app structured data (use `apify_run`)

When you need **structured data from a China app** — Douyin video search,
Xiaohongshu notes, Weibo posts, Bilibili videos, JD/Taobao product prices,
1688 wholesale listings, Xianyu second-hand, Dewu sneakers, etc. — Firecrawl
and ScrapeCreators don't cover these platforms. Use the **Apify Store** fallback
instead. Apify is a serverless scraper marketplace with hundreds of
community-maintained actors that run real browsers + proxy pools against
Chinese platforms.

**Auth:** No user-supplied key needed. sc-proxy injects the platform Apify
token automatically. The `Authorization: Bearer` header can be any fake value
— the proxy replaces it with the real token. Do NOT read `$APIFY_TOKEN` from
env, do NOT check `.env`, do NOT ask the user for an Apify key.

**When to use Apify (vs Firecrawl/ScrapeCreators):**
- ✅ China apps: 抖音, 小红书, 微博, B站, 京东, 淘宝, 1688, 闲鱼, 携程, 知乎, 豆瓣, 雪球, 优酷 (得物/快手/爱奇艺: no priced actor — search the Store)
- ✅ Southeast Asia e-commerce: Shopee, Lazada, Temu
- ✅ Instagram **stories** (ScrapeCreators has no story endpoint) — see "Instagram stories" below
- ❌ Other Western social media (TikTok/Instagram posts & reels/YouTube/X/Reddit) → use ScrapeCreators first (cheaper)
- ❌ Generic web page scraping → use Firecrawl first (cheaper)
- ❌ Hard paywall articles → use `archive_fallback` (Apify doesn't help here)

**How to pick an actor:** Use the table below first — these actors are priced in
sc-proxy (30-day successful runs in brackets). For a platform not listed, search the Store through the proxy
(`proxied_get("https://api.apify.com/v2/store?search=<platform>&limit=10")`),
prefer high 30-day run counts, and expect the default $0.007/result price.

| Platform | Actor ID | Input key |
|---|---|---|
| 抖音 search | `zen-studio~douyin-search-scraper` | `{"keywords": [...], "maxResultsPerQuery": N}` |
| 抖音 creator profile | `zen-studio~douyin-profile-scraper` (22k) | (see actor input schema) |
| 抖音 video/audio download | `apple_yang~douyin-video-audio-downloader` (5k) | (see actor input schema) |
| 抖音 transcripts | `apple_yang~douyin-transcripts-scraper` (2.6k) | (see actor input schema) |
| 小红书 search | `zen-studio~rednote-search-scraper` | `{"keywords": [...], "maxResults": N}` |
| 小红书 note detail | `sian.agency~xiaohongshu-rednote-scraper` | `{"operation": "noteDetail", "noteId": "...", "xsecToken": "..."}` |
| 小红书 data API | `socialdatax~socialdatax-xhs-data-api` (64k) | (see actor input schema) |
| 微博 hot search | `gentle_cloud~weibo-hot-search-scraper` | `{"mode": "hot_band", "includeScores": true}` |
| 微博 posts/comments | `zhorex~weibo-scraper` (5k) or `sian.agency~weibo-scraper` (17k) | (see actor input schema) |
| B站 videos/comments/danmaku | `zhorex~bilibili-scraper` (17k) | (see actor input schema) |
| 知乎 Q&A/articles | `sian.agency~zhihu-scraper` | (see actor input schema) |
| 豆瓣 reviews/subjects | `zhorex~douban-scraper` | (see actor input schema) |
| 雪球 stock discussion | `zhorex~xueqiu-scraper` (low volume) | (see actor input schema) |
| 优酷 videos | `sian.agency~youku-video-scraper` (low volume) | (see actor input schema) |
| 京东 search | `zen-studio~jd-com-search-scraper` | `{"keyword": "...", "maxProducts": N}` |
| 京东 products | `sian.agency~jd-com-product-scraper` | `{"operation": "productSearch", "keyword": "...", "maxPages": 1}` |
| 淘宝 search / detail | `zen-studio~taobao-search-scraper` / `zen-studio~taobao-detail-scraper` (18k) | (see actor input schema) |
| 淘宝/天猫 products | `sian.agency~taobao-tmall-product-scraper` | `{"operation": "keywordSearch", "keyword": "...", "maxPages": 1}` |
| 1688 wholesale | `zen-studio~1688-wholesale-scraper` or `devcake~1688-com-products-scraper` | (see actor input schema) |
| 闲鱼 search | `zen-studio~goofish-xianyu-search-scraper` | (see actor input schema) |
| Shopee / Lazada / Temu | `gio21~shopee-scraper` / `fatihtahta~lazada-scraper` / `amit123~temu-products-scraper` | (see actor input schema) |
| 携程 / Trip.com reviews | `knagymate~trip-com-reviews-scraper` (89k) | (see actor input schema) |
| Instagram stories | `data-slayer~instagram-stories-scraper` | `{"usernames": [...]}` — see "Instagram stories" |
| TikTok | `clockworks~tiktok-scraper` | `{"hashtags": [...]}` or `{"profiles": [...]}` |

Not covered by any priced actor: 快手, 爱奇艺 — search the Store and warn the user
the actor is unverified.

**Two-step Xiaohongshu workflow (search → note detail):**
The search scraper (`zen-studio~rednote-search-scraper`) returns only a
truncated `desc` (~60 chars). For the full post body, run a second call with
`sian.agency~xiaohongshu-rednote-scraper` in `noteDetail` mode, passing the
`id` and `xsec_token` from the search result row. This is the only reliable
way to get full note text for price/lodge/itinerary details.

**Usage:**

```python
import sys
sys.path.insert(0, "/data/workspace/skills/web-crawler")
from exports import apify_run

# Sync run (small batches, ≤100 results): blocks until done, returns result list
results = apify_run("zen-studio~douyin-search-scraper",
                     {"keywords": ["MacBook"], "maxResultsPerQuery": 5})
for r in results:
    print(r.get("text", "")[:80])

# To discover the right input fields for an unfamiliar actor, fetch its
# input-schema page via Firecrawl first:
from exports import scrape_markdown
schema_md = scrape_markdown("https://apify.com/<user>/<actor>/input-schema")
```

**Billing & cost control — read this before every Apify call:**

Apify uses **pay-per-event** billing: `(actor-start + result_count × per_result + add-ons) × 2` credits.
The dominant factor is **result count × per-result price**, and **each actor
prices differently** ($0.003–$0.007/result). Unknown actors default to the
highest tier ($0.007). Full pricing table and estimation examples:
`reference/apify-pricing.md`.

**You MUST estimate cost before calling:**
1. Look up the actor's per-result price (table in `reference/apify-pricing.md`;
   unlisted = $0.007).
2. Estimate result count from input params (`maxResults`, `maxProducts`, etc.).
   If no limit is set, assume 100+.
3. Calculate: `result_count × per_result × 2` = estimated credits.
4. **If estimate > 5 credits, tell the user the cost before proceeding.**

**Hard spending cap — proxy-enforced default + per-call override:**

The proxy **automatically injects** `maxTotalChargeUsd=$2.5` (≈ 5 credits) on
every actor run that doesn't already specify one. Apify terminates the run
when the budget is hit and returns whatever results were collected so far —
no overcharging, no runaway costs.

`apify_run()` also passes `max_charge_usd` (same default $2.5), which takes
precedence over the proxy default. This means:

- **Default (no extra param):** capped at ≈ 5 credits per call. Safe for
  routine searches, profile lookups, small-batch scraping.
- **Need more data?** Raise the cap explicitly: `max_charge_usd=10` (≈ 20
  credits). Use when the user explicitly wants a large dataset and you've
  already told them the estimated cost.
- **First-time test:** lower it: `max_charge_usd=0.5` (≈ 1 credit), limit
  results to 5–10, verify output quality before scaling up.
- **Disable cap entirely:** `max_charge_usd=None`. **Never do this** unless
  the user explicitly asks for an uncapped run after being warned of the
  potential cost.

**Small-batch test first:**
When using an actor for the first time, set `max_charge_usd=0.5` (≈ 1 credit)
and limit results to 5–10. Verify output quality before scaling up.

**Error handling:**
- `400 run-failed` → bad input (wrong field name). Fetch the actor's input-schema page and fix.
- `401` → proxy misconfigured (should not happen). Report to user.
- Empty result `[]` → actor ran but found nothing. Try different keywords or another actor.
- Timeout → increase `timeout` param (default 180s). Some actors are slow.

### Instagram — which route for which content

| Content | Route |
|---|---|
| Profile, bio, recent posts | `sc_get('/v1/instagram/profile', handle=...)` |
| All posts / all reels | `/v2/instagram/user/posts` / `/v1/instagram/user/reels` (paginate) |
| One post or reel | `sc_get('/v1/instagram/post', url=...)` |
| What was said in a post/reel | `/v2/instagram/media/transcript` (≤2 min) |
| Comments | `/v2/instagram/post/comments` |
| Highlights | `/v1/instagram/user/highlights` → `/v1/instagram/user/highlight/detail` |
| Reels by keyword | `/v2/instagram/reels/search` |
| **Live stories (<24h)** | `instagram_stories(handle)` (Apify `data-slayer~instagram-stories-scraper`) — section below |
| **Followers / following list** (+ public/private flag) | `instagram_followers(handle, limit)` (Apify) — SC has no endpoint, never conclude "impossible" |
| **Post likers** (+ public/private flag) | `instagram_likers(post_url, limit)` (Apify) |
| **Posts by hashtag** | `instagram_hashtag(tag, limit)` (Apify) |
| **Posts the account is tagged in** | `instagram_tagged(handle, limit)` (Apify `apify~instagram-tagged-scraper`) |
| All posts in one call | `instagram_all_posts(handle)` (SC, paginates) |
| Story views, reach/impressions, follower demographics, expired stories, DMs, private accounts | Owner-only — not available to anyone, say so |

**Full verified map (fields, costs, recipes, what is impossible):**
`reference/instagram.md` — read it before any Instagram analysis.

`/v1/instagram/song/reels` is deprecated upstream (Instagram removed audio
pages); don't call it.

### Instagram stories (use `apify_run`)

Story URLs (`instagram.com/stories/<user>/<id>`) are not reachable via
ScrapeCreators (no story endpoint; only highlights) or Firecrawl (login wall).
Don't retry those — call Apify directly through the proxy (no user key):

```python
from exports import apify_run
items = apify_run("data-slayer~instagram-stories-scraper",
                  {"usernames": ["rafaelkborges"]}, max_charge_usd=0.5)
story = next(i for i in items if str(i.get("id")) == "4002796791460342130")
video_url = story["video_versions"][0]["url"]   # image-only stories: image_versions2
```

- Covers **public accounts, active stories only** (< 24h). Accounts with none
  return a row `{"status": "no_active_stories"}`. Private or expired stories
  can't be fetched by anyone — say so.
- Match the story by **`id`** (the number in the URL). `pk` is null.
- CDN links expire quickly — download the mp4 right away.
- To get what was said, pass the mp4 to the video-analysis skill (native mode
  handles a ~12 MB / 60 s story in one call; transcript + on-screen text).
- If Apify returns nothing: check `/v1/instagram/user/highlights`; else if the
  user's Chrome is connected (control-browser) and logged into Instagram, open
  the story there; else ask for a screen recording (not a screenshot — most
  stories are spoken video).
- Typical: ~8–13 s per batch of accounts, ~$0.0025/result.

**Cost discipline:** Apify actors are more expensive than Firecrawl/ScrapeCreators.
Only use Apify when the cheaper options can't get the data (China apps,
Instagram stories, structured e-commerce fields). For a single web page, always try Firecrawl first.

## What each service is for
### ScrapeCreators — Social media data extraction (27+ platforms)

Use for any request involving social media profiles, posts, videos, comments, transcripts, search, ads, trending content, or engagement metrics. Covers TikTok, Instagram, YouTube, LinkedIn, Facebook, Twitter/X, Reddit, Threads, Bluesky, Pinterest, Snapchat, Twitch, Kick, Truth Social, TikTok Shop, Google search, and link-in-bio services (Linktree, Komi, Pillar, Linkbio, Linkme, Amazon Shop).

**Base URL:** `https://api.scrapecreators.com`
**Auth:** No user-supplied key needed. sc-proxy injects platform credentials automatically — just send the request. The `x-api-key` header can be any value or omitted entirely. Do NOT bail out or ask the user for a key if `$SCRAPECREATORS_API_KEY` looks unset; that env var is intentionally not required.
**Method:** All endpoints use GET requests with query params. Responses are JSON.

### Firecrawl — Fallback web page scraper

Only a fallback crawler for one web page when ordinary fetching fails. Use `POST /v2/scrape` with a single `url` and focused formats like `markdown`, `html`, `rawHtml`, `links`, `summary`, or constrained `json`/`question`/`highlights` extraction.

**Auth:** No user-supplied key needed. sc-proxy injects the Firecrawl credential automatically when you call through `core.http_client.proxied_post` — just send the request. Do NOT read `$FIRECRAWL_API_KEY` from env, do NOT check `.env` for it, and do NOT ask the user for a Firecrawl key if it looks unset; that env var is intentionally not required. The same proxy-injection model as ScrapeCreators applies here.

Do not use Firecrawl crawl/map/search/agent/browser endpoints. Do not request screenshots, audio, branding, images, or browser actions unless the proxy policy is expanded later.

---

## ScrapeCreators — Intent routing
Map user intent to the right endpoint. Endpoint paths use the pattern `/v1/platform/action`.

**Important:** After selecting an endpoint from the tables below, fetch its OpenAPI spec at `https://docs.scrapecreators.com/{path}/openapi.json` for full parameter details, types, and example response before making the actual API call. For example: `https://docs.scrapecreators.com/v1/tiktok/profile/openapi.json`

### Profiles / User Info
| Platform | Endpoint | Primary Param | Example |
|----------|----------|---------------|---------|
| TikTok | `/v1/tiktok/profile` | handle | `stoolpresidente` |
| Instagram | `/v1/instagram/profile` | handle | `jane` |
| YouTube | `/v1/youtube/channel` | handle, channelId, or url | `ThePatMcAfeeShow` |
| LinkedIn (person) | `/v1/linkedin/profile` | url | `https://www.linkedin.com/in/parrsam/` |
| LinkedIn (company) | `/v1/linkedin/company` | url | `https://linkedin.com/company/shopify` |
| Facebook | `/v1/facebook/profile` | url | `https://www.facebook.com/mantraindianfolsom` |
| Twitter/X | `/v1/twitter/profile` | handle | `elonmusk` |
| Reddit | `/v1/reddit/subreddit/details` | subreddit or url | `AskReddit` |
| Threads | `/v1/threads/profile` | handle | `zuck` |
| Bluesky | `/v1/bluesky/profile` | handle | `jay.bsky.team` |
| Pinterest | `/v1/pinterest/user/boards` | handle | `pinterest` |
| Truth Social | `/v1/truthsocial/profile` | handle | `realDonaldTrump` |
| Twitch | `/v1/twitch/profile` | handle | `ninja` |
| Snapchat | `/v1/snapchat/profile` | handle | `djkhaled` |

### Posts / Content Feeds
| Platform | Endpoint | Primary Param | Example |
|----------|----------|---------------|---------|
| TikTok videos | `/v3/tiktok/profile/videos` | handle | `stoolpresidente` |
| Instagram posts | `/v2/instagram/user/posts` | handle | `jane` |
| Instagram reels | `/v1/instagram/user/reels` | handle or user_id | `jane` or `2700692569` |
| Instagram highlights | `/v1/instagram/user/highlights` | handle or user_id | `jane` or `2700692569` |
| YouTube videos | `/v1/youtube/channel/videos` | handle or channelId | `ThePatMcAfeeShow` |
| YouTube shorts | `/v1/youtube/channel/shorts` | handle or channelId | `starterstory` |
| YouTube playlist | `/v1/youtube/playlist` | playlist_id | `PLP32wGpgzmIlInfgKVFfCwVsxgGqZNIiS` |
| LinkedIn posts | `/v1/linkedin/company/posts` | url | `https://linkedin.com/company/shopify` |
| Facebook posts | `/v1/facebook/profile/posts` | url or pageId | `https://www.facebook.com/pacemorby` |
| Facebook reels | `/v1/facebook/profile/reels` | url | `https://www.facebook.com/Spurs` |
| Facebook photos | `/v1/facebook/profile/photos` | url | `https://www.facebook.com/Spurs` |
| Facebook group posts | `/v1/facebook/group/posts` | url or group_id | `742354120555345` |
| Twitter tweets | `/v1/twitter/user/tweets` | handle | `elonmusk` |
| Reddit posts | `/v1/reddit/subreddit` | subreddit | `AskReddit` |
| Threads posts | `/v1/threads/user/posts` | handle | `zuck` |
| Bluesky posts | `/v1/bluesky/user/posts` | handle or user_id | `jay.bsky.team` |
| Truth Social posts | `/v1/truthsocial/user/posts` | handle or user_id | `realDonaldTrump` |
| Pinterest board | `/v1/pinterest/board` | url | `https://www.pinterest.com/...` |

### Single Post / Video Details
| Platform | Endpoint | Primary Param | Example |
|----------|----------|---------------|---------|
| TikTok | `/v2/tiktok/video` | url | `https://www.tiktok.com/@randomspamvideos25/video/7251387037834595630` |
| Instagram | `/v1/instagram/post` | url | `https://www.instagram.com/reel/DOq6eV6iIgD` |
| Instagram highlight | `/v1/instagram/user/highlight/detail` | id | `18067016518767507` |
| YouTube | `/v1/youtube/video` | url | `https://www.youtube.com/watch?v=Y2Ah_DFr8cw` |
| YouTube community post | `/v1/youtube/community-post` | url | `https://www.youtube.com/post/Ugkxvj2KoApYAXoqLWnKVr6zZe5JjeHrQeP8` |
| LinkedIn | `/v1/linkedin/post` | url | `https://www.linkedin.com/pulse/being-father-has-made-me-better-leader...` |
| Facebook | `/v1/facebook/post` | url | `https://www.facebook.com/reel/1535656380759655` |
| Twitter/X | `/v1/twitter/tweet` | url | `https://twitter.com/elonmusk/status/...` |
| Twitter/X community | `/v1/twitter/community` | url | `https://twitter.com/i/communities/...` |
| Twitter/X community tweets | `/v1/twitter/community/tweets` | url | `https://twitter.com/i/communities/...` |
| Reddit | `/v1/reddit/post/comments` | url | `https://www.reddit.com/r/AskReddit/comments/...` |
| Threads | `/v1/threads/post` | url | `https://www.threads.net/@zuck/post/...` |
| Bluesky | `/v1/bluesky/post` | url | `https://bsky.app/profile/.../post/...` |
| Truth Social | `/v1/truthsocial/post` | url | `https://truthsocial.com/@realDonaldTrump/posts/...` |
| Pinterest | `/v1/pinterest/pin` | url | `https://www.pinterest.com/pin/...` |
| Twitch clip | `/v1/twitch/clip` | url | `https://clips.twitch.tv/...` |
| Kick clip | `/v1/kick/clip` | url | `https://kick.com/...` |

### Comments
| Platform | Endpoint | Primary Param | Example |
|----------|----------|---------------|---------|
| TikTok | `/v1/tiktok/video/comments` | url | `https://www.tiktok.com/@stoolpresidente/video/7499229683859426602` |
| Instagram | `/v2/instagram/post/comments` | url | `https://www.instagram.com/reel/DOq6eV6iIgD` |
| YouTube | `/v1/youtube/video/comments` | url | `https://www.youtube.com/watch?v=dQw4w9WgXcQ` |
| Facebook | `/v1/facebook/post/comments` | url or feedback_id | `https://www.facebook.com/reel/753347914167361` |
| Reddit | `/v1/reddit/post/comments` | url | `https://www.reddit.com/r/AskReddit/comments/...` |

### Transcripts

⚠️ **Metadata first.** A transcript endpoint returns **speech text only — no
speaker, no title, no uploader**. When summarizing a video, call the video
metadata endpoint FIRST (`youtube_video` / `/v1/tiktok/video` / …) for title,
description and uploader, then the transcript. **Never attribute a speaker or
public figure from transcript content alone** — if metadata is unavailable, say
"speaker unidentified" and summarize without naming anyone.

| Platform | Endpoint | Example | Note |
|----------|----------|---------|------|
| TikTok | `/v1/tiktok/video/transcript` | `url=https://www.tiktok.com/...&lang=en` | also via `/v2/tiktok/video` with get_transcript=true |
| Instagram | `/v2/instagram/media/transcript` | `url=https://www.instagram.com/reel/...` | AI-powered, 10-30s, under 2min |
| YouTube | `/v1/youtube/video/transcript` | `url=https://www.youtube.com/watch?v=bjVIDXPP7Uk` | also included in `/v1/youtube/video` response |
| Facebook | `/v1/facebook/post/transcript` | `url=https://www.facebook.com/reel/...` | under 2min only |
| Twitter/X | `/v1/twitter/tweet/transcript` | `url=https://twitter.com/...` | AI-powered, slow |

### Search
| Platform | Endpoint | Primary Param | Example |
|----------|----------|---------------|---------|
| TikTok users | `/v1/tiktok/search/users` | query | `funny` |
| TikTok videos (keyword) | `/v1/tiktok/search/keyword` | query | `funny` |
| TikTok videos (hashtag) | `/v1/tiktok/search/hashtag` | hashtag | `fyp` |
| TikTok top (photos+videos) | `/v1/tiktok/search/top` | query | `funny` |
| Instagram reels | `/v2/instagram/reels/search` | query | `dogs` |
| YouTube | `/v1/youtube/search` | query | `funny` |
| YouTube hashtag | `/v1/youtube/search/hashtag` | hashtag | `funny` |
| Reddit (all) | `/v1/reddit/search` | query | `best programming languages` |
| Reddit (in subreddit) | `/v1/reddit/subreddit/search` | subreddit + query | `AskReddit` + `funny` |
| Threads posts | `/v1/threads/search` | query | `AI` |
| Threads users | `/v1/threads/search/users` | query | `zuck` |
| Pinterest | `/v1/pinterest/search` | query | `home decor` |
| Google | `/v1/google/search` | query | `best restaurants in NYC` |

### Ad Libraries
| Platform | Endpoint | Primary Param | Example |
|----------|----------|---------------|---------|
| Facebook ads search | `/v1/facebook/adLibrary/search/ads` | query | `running` |
| Facebook company ads | `/v1/facebook/adLibrary/company/ads` | pageId or companyName | `Lululemon` |
| Facebook ad detail | `/v1/facebook/adLibrary/ad` | id or url | `702369045530963` |
| Facebook find companies | `/v1/facebook/adLibrary/search/companies` | query | `Nike` |
| Google company ads | `/v1/google/company/ads` | domain or advertiser_id | `nike.com` |
| Google ad detail | `/v1/google/ad` | url | `https://adstransparency.google.com/...` |
| Google find advertisers | `/v1/google/adLibrary/advertisers/search` | query | `Nike` |
| LinkedIn ads search | `/v1/linkedin/ads/search` | company or keyword | `Shopify` |
| LinkedIn ad detail | `/v1/linkedin/ad` | url | `https://www.linkedin.com/ad/...` |
| Reddit ads search | `/v1/reddit/ads/search` | query | `gaming` |
| Reddit ad detail | `/v1/reddit/ad` | id | `t3_abc123` |

### Trending / Popular
| Content | Endpoint | Param | Example |
|---------|----------|-------|---------|
| Trending feed | `/v1/tiktok/get-trending-feed` | region (required) | `US` |
| Popular videos | `/v1/tiktok/videos/popular` | | |
| Popular creators | `/v1/tiktok/creators/popular` | | |
| Popular hashtags | `/v1/tiktok/hashtags/popular` | | |
| Popular songs | `/v1/tiktok/songs/popular` | | |
| Song details | `/v1/tiktok/song` | clipId | `7439295283975702544` |
| Videos using song | `/v1/tiktok/song/videos` | clipId | `7439295283975702544` |
| Trending shorts (YT) | `/v1/youtube/shorts/trending` | | |

### Followers / Following / Live (TikTok only)
| Type | Endpoint | Example |
|------|----------|---------|
| Following | `/v1/tiktok/user/following` | `handle=stoolpresidente` |
| Followers | `/v1/tiktok/user/followers` | `handle=stoolpresidente` |
| Audience demographics | `/v1/tiktok/user/audience` (26 credits!) | `handle=shakira` |
| Live stream | `/v1/tiktok/user/live` | `handle=thejustalex` |

### TikTok Shop
| Type | Endpoint | Primary Param | Example |
|------|----------|---------------|---------|
| Search products | `/v1/tiktok/shop/search` | query | `shoes` |
| Store products | `/v1/tiktok/shop/products` | url | `https://www.tiktok.com/shop/store/goli-nutrition/7495794203056835079` |
| Product detail | `/v1/tiktok/product` | url | `https://www.tiktok.com/shop/pdp/goli-ashwagandha-gummies.../1729587769570529799` |
| Product reviews | `/v1/tiktok/shop/product/reviews` | url or product_id | `1731578642912612516` |
| User showcase | `/v1/tiktok/user/showcase` | handle | `mrtiktokreviews` |

### Link-in-Bio / Other
| Service | Endpoint | Param | Example |
|---------|----------|-------|---------|
| Linktree | `/v1/linktree` | url | `https://linktr.ee/...` |
| Komi | `/v1/komi` | url | `https://komi.io/...` |
| Pillar | `/v1/pillar` | url | `https://pillar.io/...` |
| Linkbio | `/v1/linkbio` | url | `https://linkbio.co/...` |
| Linkme | `/v1/linkme` | url | `https://linkme.bio/...` |
| Amazon Shop | `/v1/amazon/shop` | url | `https://www.amazon.com/shop/...` |
| Instagram basic profile | `/v1/instagram/basic-profile` | userId | `314216` |
| Instagram embed HTML | `/v1/instagram/user/embed` | handle | `jane` |
| Age/Gender detect | `/v1/detect/age-gender` | url (social profile) | `https://www.tiktok.com/@charlidamelio` |
| Credit balance | `/v1/credit/balance` | (none) | |

### ScrapeCreators pagination

Paginated endpoints return a cursor/token in the response. Pass it back as a query param to get the next page.

| Cursor Field | Used By |
|-------------|---------|
| `cursor` | TikTok comments/search/song videos, Instagram comments, Reddit subreddit search, Pinterest, Bluesky, Facebook reels/photos/posts/comments, TikTok Shop products/user showcase |
| `max_cursor` | TikTok profile videos |
| `min_time` | TikTok following/followers |
| `continuationToken` | YouTube (all paginated endpoints) |
| `after` | Reddit posts, Reddit search |
| `next_max_id` | Instagram posts, Truth Social posts |
| `max_id` | Instagram reels |
| `page` | TikTok popular/shop, Instagram reels search, LinkedIn company posts, TikTok Shop reviews |
| `paginationToken` | LinkedIn ads |

### ScrapeCreators known limitations

- **Handles**: pass without the `@` symbol. Use `charlidamelio` not `@charlidamelio`. Applies to TikTok, Instagram, Twitter, Threads, Bluesky, Snapchat, Twitch, Pinterest, Truth Social
- **YouTube handles**: pass without the `@` symbol. Use `ThePatMcAfeeShow` not `@ThePatMcAfeeShow`. You can also pass a channelId or full URL instead
- **Hashtags**: pass without the `#` symbol. Use `fyp` not `#fyp`. Applies to TikTok and YouTube hashtag search endpoints
- **Twitter**: returns ~100 most popular tweets, not chronological/latest
- **Threads**: only last 20-30 posts visible publicly
- **Facebook posts**: only 3 posts per page (API limitation)
- **Facebook group posts**: only 3 posts per page (same limitation)
- **LinkedIn company posts**: max 7 pages
- **Instagram play counts**: IG-only views (excludes cross-posted FB views)
- **Truth Social**: only prominent users (Trump, Vance, etc.) work publicly
- **Transcripts**: all transcript endpoints require video under 2 minutes
- **Reddit subreddit names**: case-sensitive! Use "AskReddit" not "askreddit"

---

## Access patterns
### ScrapeCreators (social media)

Use Python with `core.http_client.proxied_get` so sc-proxy injects credentials and bills correctly. Include a typed `SC-CALLER-ID` header (`chat:`, `job:`, `preview:`, etc.) for cost tracking. **Do not read `$SCRAPECREATORS_API_KEY` from env and do not ask the user for a key — the proxy handles it.**

```python
from core.http_client import proxied_get

headers = {"SC-CALLER-ID": "chat:youtube-transcript"}

transcript = proxied_get(
    "https://api.scrapecreators.com/v1/youtube/video/transcript",
    params={"url": "https://www.youtube.com/watch?v=VIDEO_ID", "language": "en"},
    headers=headers,
    timeout=20,
).json()
```

Bash/curl works too (proxy is transparent), but Python is preferred for cost tracking:

```bash
curl -s "https://api.scrapecreators.com/v1/tiktok/profile?handle=charlidamelio" \
  -H "x-api-key: any"
```

Each endpoint has its own OpenAPI spec at `https://docs.scrapecreators.com/{path}/openapi.json`. **Always fetch the per-endpoint spec first** to get full parameter details before making the actual API call. The full spec is at `https://docs.scrapecreators.com/openapi.json` (large file — prefer per-endpoint specs).

Common optional params:
- **`trim`** (boolean): reduces response payload size. Use when you only need key metrics.
- **`region`** (string): 2-letter country code for proxy location. Does NOT filter by region — just routes through that country's proxy.

### Firecrawl (web page fallback via transparent proxy)

No Firecrawl API key required — sc-proxy injects it. Don't look it up in `.env` or ask the user. Just call:

```python
from core.http_client import proxied_post

headers = {"SC-CALLER-ID": "chat:web-crawl-fallback"}

page = proxied_post(
    "https://api.firecrawl.dev/v2/scrape",
    json={
        "url": "https://example.com/article",
        "formats": ["markdown", "links"],
        "onlyMainContent": True,
        "timeout": 60000,
    },
    headers=headers,
).json()
```

## Decision rules
Route every request to the right backend. The user should never need to specify which API to use.

### Social media request (profile, posts, comments, search, ads, trending, transcripts)
Use **ScrapeCreators**. Match the user's intent to an endpoint from the routing tables above. Strip `@` from handles and `#` from hashtags before calling. Fetch the per-endpoint OpenAPI spec first for full param details.

### YouTube URL or YouTube content request
Use **ScrapeCreators** for YouTube (channel info, videos, shorts, playlists, comments, transcripts, search, trending shorts). Match the user's intent to the appropriate YouTube endpoint from the routing tables above.

For a YouTube URL, use `/v1/youtube/video` to get video details and transcript. If the user's goal is content analysis, summarization, quote extraction, or topic mining, the transcript is included in the video response.

For a YouTube topic query, use `/v1/youtube/search` to find relevant videos, then call `/v1/youtube/video` only for the videos needed. Avoid fetching many videos by default.

### Blocked or JS-heavy web page
Use **Firecrawl** once with `formats:["markdown","links"]` and `onlyMainContent:true`. Treat the returned Markdown as the extraction substrate, not as final truth: parse the title, price/value fields, specs, body description, image URLs, o

…(truncated)
