Web Crawler

Web scraping plus social data: YouTube, TikTok, Instagram (incl. stories, followers, likers, hashtags), LinkedIn, Reddit, Threads, plus China apps and Asian e-commerce via Apify (Douyin, Xiaohongshu, Weibo, Bilibili, Zhihu, Douban, Xueqiu, Taobao, JD, 1688, Xianyu, Shopee, Lazada, Temu, Trip.com reviews), plus robust web-page fallback extraction. Use when extracting public posts, transcripts, or pages JS-heavy enough to block plain fetch (e.g. download YouTube transcript, scrape TikTok comments, listing pages behind anti-bot/Cloudflare). Auto-fallback trigger terms: web_fetch failed, 403, anti-bot, cloudflare, JS-heavy page. Also use for whole-site link audits and "this button/link does not work" reports (check all links, broken links, 404, button not working) — you investigate with site-agnostic fetch / browse tools.

starchild-ai-agent Updated 33 repo stars

File contents

starchild-ai-agent/official-skills commit 62299f150d

Frequently asked questions

npx get-skill add starchild-ai-agent/web-crawler