Always check the catalog freshness before recommending. If stale (> N days), surface that to the user.
The headline use case is Chinese social/e-commerce platforms. When the user names one (小红书/抖音/B站/微博/知乎/贴吧/快手/公众号/视频号/淘宝/京东/拼多多/豆瓣/雪球…), route directly to the fast path (workflows/match-and-crawl.md Step 3B) via references/chinese-social-platforms.md — never show the generic category menu. The general five-category catalog is the fallback breadth for every other target.
Compliance gate (non-negotiable): most Chinese-platform tools are reverse-engineered and violate platform ToS. Always show the compliance reminder before tool selection, re-confirm intent before the first network request, never help evade anti-bot / risk-control systems, and default to low request rates + anonymized personal data.
| Tag |
Means |
web-scraper |
Static HTML / simple HTTP fetch (BeautifulSoup, httpx, Selectolax) |
dynamic-scraper |
JS-rendered pages, SPAs (Playwright, Selenium, Crawl4AI) |
api-collector |
REST/GraphQL endpoints, SDK-driven pulls, ETL pipelines |
agent-skill |
Claude/GPT agent skills, MCP servers, tool-use frameworks |
dataset |
Public datasets, awesome-lists, curated resource repos |
Categories drive both discovery keywords and the matching menu.
- "抓小红书 / 抖音评论 / 微博热搜…" (Chinese platform named) — fast path: platform-tagged shortlist + compliance gate, straight to a tool.
- "Find/discover tools" — refresh the catalog, search GitHub for collection-class repos.
- "I want to scrape/fetch X" (any other target) — match a tool from the catalog to a target and start crawling.
- "Browse the catalog" — show categories / tool cards without scraping.
- "Schedule / automate" — set up the periodic refresh hook.
If the message is ambiguous, ask one minimal question to disambiguate, then route.
| User says |
Route to |
any Chinese platform name (小红书/抖音/B站/微博/淘宝…, full alias table in references/chinese-social-platforms.md) |
workflows/match-and-crawl.md (Step 3B fast path) |
| "refresh", "discover", "find new repos", "update catalog" |
workflows/discover-catalog.md |
| "I want to scrape/fetch/collect X", "抓 X 数据" |
workflows/match-and-crawl.md |
| "show me the catalog", "what tools do we have", "browse" |
workflows/browse-catalog.md |
| "schedule", "automate refresh", "定时", "cron" |
workflows/schedule-refresh.md |
After routing, the workflow tells you which references to load. Do not preload everything.
tool-catalog.json — canonical structured catalog (data source)tool-catalog.md — human-readable catalog (generated, do not edit)discovery-log.md — append-only run logcategory-keywords.md — search keywords + GitHub topic mapping per categorychinese-social-platforms.md — canonical platform registry: alias → key map, tag conventions, per-platform crawl characteristics (load whenever a Chinese platform is suspected)repo-schema.md — the JSON schema every catalog entry must satisfyrate-limit-guide.md — GitHub API quota, pagination, retry/backoff patternsllm-judging.md — optional LLM judge mechanism + safe-pruning guards (load only when running/tuning discovery)
| Workflow |
Purpose |
discover-catalog.md |
Scan GitHub, dedupe, update catalog JSON + MD |
match-and-crawl.md |
Progressive-disclosure matching → tool selection → crawl |
browse-catalog.md |
Read-only category/card view, no network crawling |
schedule-refresh.md |
Install a periodic refresh hook (cron / Task Scheduler) |
tools/*.md |
Per-tool crawl workflows — loaded by match-and-crawl.md Step 5 when a catalog entry's workflow_file points here. Each is a concrete crawl recipe (spec → skeleton → pre-flight → run → validate → report). See tools/mediacrawler.md for the compliance-gated pattern. |
Existing per-tool workflows: tools/scrapy.md, tools/playwright.md, tools/crawl4ai.md, tools/mediacrawler.md.
1---2name: collection-skill3description: Chinese-social-media crawler picker — recommends the right tool and actually fetches posts, notes, comments, and reviews from 小红书 / 抖音 / 哔哩哔哩 / 微博 / 知乎 / 贴吧 / 快手 / 微信公众号 / 视频号 / 淘宝 / 京东 / 拼多多 / 豆瓣 / 雪球 (xhs, douyin, bilibili, weibo, zhihu, tieba, kuaishou, wechat, taobao, jd, pdd, douban, xueqiu), behind a compliance gate. Also covers general scraping of any website or API (抓取 / 采集 / 爬取 / 爬虫 / 抓数据 / 获取数据 / 数据采集 / scrape / crawl / collect data). Maintains a curated, auto-refreshed catalog of scrapers, API collectors, MCP/agent skills, and datasets.4---56<essential_principles>7<principle name="Catalog First, Crawl Second">8This skill has two halves that share one knowledge base:9- **Discover & catalog** — periodically scan GitHub for "collection-class" repos (scrapers, API collectors, agent skills/MCP, datasets/awesome-lists) and store them in `references/tool-catalog.{md,json}`.10- **Match & crawl** — when the user names a target, recommend tools from the catalog via progressive disclosure, then start crawling.1112Always check the catalog freshness before recommending. If stale (> N days), surface that to the user.13</principle>14<principle name="Chinese-Social First">15The headline use case is Chinese social/e-commerce platforms. When the user names one (小红书/抖音/B站/微博/知乎/贴吧/快手/公众号/视频号/淘宝/京东/拼多多/豆瓣/雪球…), route **directly** to the fast path (`workflows/match-and-crawl.md` Step 3B) via `references/chinese-social-platforms.md` — never show the generic category menu. The general five-category catalog is the fallback breadth for every other target.1617**Compliance gate (non-negotiable):** most Chinese-platform tools are reverse-engineered and violate platform ToS. Always show the compliance reminder before tool selection, re-confirm intent before the first network request, never help evade anti-bot / risk-control systems, and default to low request rates + anonymized personal data.18</principle>1920<principle name="Progressive Disclosure">21Never dump the whole catalog. Show a **category menu** first, then drill into tool cards within the chosen category, then load the matching workflow only after the user picks a tool. Load references lazily — only what the current step needs.22</principle>2324<principle name="Five Canonical Categories">25Every repo in the catalog is tagged with exactly one primary category:2627| Tag | Means |28|-----|-------|29| `web-scraper` | Static HTML / simple HTTP fetch (BeautifulSoup, httpx, Selectolax) |30| `dynamic-scraper` | JS-rendered pages, SPAs (Playwright, Selenium, Crawl4AI) |31| `api-collector` | REST/GraphQL endpoints, SDK-driven pulls, ETL pipelines |32| `agent-skill` | Claude/GPT agent skills, MCP servers, tool-use frameworks |33| `dataset` | Public datasets, awesome-lists, curated resource repos |3435Categories drive both discovery keywords and the matching menu.36</principle>3738<principle name="Catalog Is the Source of Truth">39- `references/tool-catalog.json` — structured data, the canonical source.40- `references/tool-catalog.md` — human-readable view, **regenerated from JSON** by `scripts/build_catalog_md.py`. Never hand-edit the markdown.41- `references/discovery-log.md` — append-only run history (when, what, how many, errors).42</principle>4344<principle name="Safety & Boundaries">45- Respect `robots.txt`, rate limits, and ToS. Default to authenticated GitHub API calls (higher limits) when scanning repos.46- Never store credentials in the repo. Read tokens from env vars (`GITHUB_TOKEN`) or the `gh` CLI keyring.47- Always confirm scope with the user before the first network request against a new target domain.48</principle>4950<principle name="LLM Judging Is Optional & Safely Guarded">51Discovery *can* use an LLM (`LLM_API_KEY`) to judge candidates and reassign categories, but **pruning is never the model's call alone** — deterministic collection-signal and human-curation guards defend it. Full mechanism in `references/llm-judging.md`. Not needed for the match-and-crawl path.52</principle>53</essential_principles>5455<intake>56On invocation, determine the user's intent. Most messages fall into one of:57581. **"抓小红书 / 抖音评论 / 微博热搜…" (Chinese platform named)** — fast path: platform-tagged shortlist + compliance gate, straight to a tool.592. **"Find/discover tools"** — refresh the catalog, search GitHub for collection-class repos.603. **"I want to scrape/fetch X" (any other target)** — match a tool from the catalog to a target and start crawling.614. **"Browse the catalog"** — show categories / tool cards without scraping.625. **"Schedule / automate"** — set up the periodic refresh hook.6364If the message is ambiguous, ask **one** minimal question to disambiguate, then route.65</intake>6667<routing>68Map user intent to a workflow:6970| User says | Route to |71|-----------|----------|72| any Chinese platform name (小红书/抖音/B站/微博/淘宝…, full alias table in `references/chinese-social-platforms.md`) | `workflows/match-and-crawl.md` (Step 3B fast path) |73| "refresh", "discover", "find new repos", "update catalog" | `workflows/discover-catalog.md` |74| "I want to scrape/fetch/collect X", "抓 X 数据" | `workflows/match-and-crawl.md` |75| "show me the catalog", "what tools do we have", "browse" | `workflows/browse-catalog.md` |76| "schedule", "automate refresh", "定时", "cron" | `workflows/schedule-refresh.md` |7778After routing, the workflow tells you which references to load. Do not preload everything.79</routing>8081<reference_index>82All in `references/`:8384- `tool-catalog.json` — canonical structured catalog (data source)85- `tool-catalog.md` — human-readable catalog (generated, do not edit)86- `discovery-log.md` — append-only run log87- `category-keywords.md` — search keywords + GitHub topic mapping per category88- `chinese-social-platforms.md` — canonical platform registry: alias → key map, tag conventions, per-platform crawl characteristics (load whenever a Chinese platform is suspected)89- `repo-schema.md` — the JSON schema every catalog entry must satisfy90- `rate-limit-guide.md` — GitHub API quota, pagination, retry/backoff patterns91- `llm-judging.md` — optional LLM judge mechanism + safe-pruning guards (load only when running/tuning discovery)92</reference_index>9394<workflows_index>95All in `workflows/`:9697| Workflow | Purpose |98|----------|---------|99| `discover-catalog.md` | Scan GitHub, dedupe, update catalog JSON + MD |100| `match-and-crawl.md` | Progressive-disclosure matching → tool selection → crawl |101| `browse-catalog.md` | Read-only category/card view, no network crawling |102| `schedule-refresh.md` | Install a periodic refresh hook (cron / Task Scheduler) |103| `tools/*.md` | **Per-tool crawl workflows** — loaded by `match-and-crawl.md` Step 5 when a catalog entry's `workflow_file` points here. Each is a concrete crawl recipe (spec → skeleton → pre-flight → run → validate → report). See `tools/mediacrawler.md` for the compliance-gated pattern. |104105Existing per-tool workflows: `tools/scrapy.md`, `tools/playwright.md`, `tools/crawl4ai.md`, `tools/mediacrawler.md`.106</workflows_index>107108<success_criteria>109This skill works when:110- A Chinese-platform request hits the fast path and sees the compliance reminder before any tool card.111- The catalog contains real, recently-verified entries across all five categories.112- `tool-catalog.md` and `tool-catalog.json` stay in sync (MD regenerated from JSON).113- A user asking "I want to scrape X" gets a category menu → tool card → workflow in ≤ 2 turns.114- Discovery runs are idempotent and logged in `discovery-log.md`.115- No credentials are committed; tokens come from env or `gh` keyring.116</success_criteria>
Yuuqq