Social Media Crawling
Downloads and extracts content from social media platforms.
Supported Platforms and URL Patterns
| Platform |
URL Pattern Examples |
| Bilibili |
bilibili.com/video/BVxxxx, b23.tv/xxxxx |
| YouTube |
youtube.com/watch?v=xxx, youtu.be/xxx |
| TikTok |
tiktok.com/@user/video/xxx, vm.tiktok.com/xxx |
| Douyin |
douyin.com/video/xxx, v.douyin.com/xxx |
| Xiaohongshu |
xiaohongshu.com/explore/xxx, xhslink.com/xxx |
| Kuaishou |
kuaishou.com/short-video/xxx, v.kuaishou.com/xxx |
Supported Content Types
- Videos: Short videos, long videos from all platforms
- Images: Xiaohongshu image posts, Douyin image posts
- Metadata: Title, description, tags, cover image
When to Use
Use this skill when user:
- Provides a social media URL matching the patterns above
- Wants to download/save videos from social platforms
- Wants to extract content for re-publishing
- Needs video metadata (title, description, tags)
- Mentions keywords like: download, save, crawl, extract, 下载, 抓取, 爬取
Do NOT use when:
- User only wants to view/preview content (no download needed)
- URL is not from a supported platform
Workflow
Step 1: Get Social Media URL
Obtain the post URL from user.
Step 2: Submit Crawl Task
Call createCrawlTask with the link.
Step 3: Poll for Results
- Wait 30 seconds between status checks
- Call
getCrawlTaskStatus to check progress
Step 4: Return Results
On success, return:
- Video/media URLs
- Title
- Description
- Tags
Examples
Example 1: Download Bilibili Video
User request: "Download this Bilibili video: https://www.bilibili.com/video/BV1xx..."
1. createCrawlTask:
- link: "https://www.bilibili.com/video/BV1xx..."
2. Wait 30 seconds
3. Poll getCrawlTaskStatus until success
4. Return video URL, title, description, tags
Example 2: Download from Short Link
User request: "下载这个视频 https://v.douyin.com/xxx"
1. createCrawlTask:
- link: "https://v.douyin.com/xxx"
2. Wait 30 seconds
3. Poll getCrawlTaskStatus until success
4. Return video URL, title, description, tags
Example 3: Extract for Re-publishing
1. createCrawlTask with source URL
2. Poll getCrawlTaskStatus until completed
3. Extract media URLs, title, description, tags
4. Use content skill to save media
5. Use publish skill to publish to target platforms
Task Status Values
| Status |
Description |
| pending |
Task created, waiting to start |
| running |
Task is actively processing |
| success |
Task completed successfully |
| failed |
Task failed with error |
Important Notes
- Processing time varies by platform and content size
- Large videos may take longer to download
- Some platforms may have rate limits
- Media URLs are temporary - save to content library for permanent storage
1---2name: crawling-social-media3description: Downloads and extracts content from social media platforms. Use when user wants to crawl, download, or extract videos/content from Bilibili, YouTube, TikTok, Douyin, Instagram, Twitter, Xiaohongshu. 下载视频、抓取视频、爬取内容、提取视频、下载B站视频、下载抖音视频、下载YouTube视频、视频采集。4---56# Social Media Crawling78Downloads and extracts content from social media platforms.910## Supported Platforms and URL Patterns1112| Platform | URL Pattern Examples |13| ----------- | ---------------------------------------------------- |14| Bilibili | `bilibili.com/video/BVxxxx`, `b23.tv/xxxxx` |15| YouTube | `youtube.com/watch?v=xxx`, `youtu.be/xxx` |16| TikTok | `tiktok.com/@user/video/xxx`, `vm.tiktok.com/xxx` |17| Douyin | `douyin.com/video/xxx`, `v.douyin.com/xxx` |18| Xiaohongshu | `xiaohongshu.com/explore/xxx`, `xhslink.com/xxx` |19| Kuaishou | `kuaishou.com/short-video/xxx`, `v.kuaishou.com/xxx` |2021## Supported Content Types2223- **Videos**: Short videos, long videos from all platforms24- **Images**: Xiaohongshu image posts, Douyin image posts25- **Metadata**: Title, description, tags, cover image2627## When to Use2829Use this skill when user:3031- Provides a social media URL matching the patterns above32- Wants to download/save videos from social platforms33- Wants to extract content for re-publishing34- Needs video metadata (title, description, tags)35- Mentions keywords like: download, save, crawl, extract, 下载, 抓取, 爬取3637**Do NOT use** when:3839- User only wants to view/preview content (no download needed)40- URL is not from a supported platform4142## Workflow4344### Step 1: Get Social Media URL4546Obtain the post URL from user.4748### Step 2: Submit Crawl Task4950Call `createCrawlTask` with the link.5152### Step 3: Poll for Results53541. Wait **30 seconds** between status checks552. Call `getCrawlTaskStatus` to check progress5657### Step 4: Return Results5859On success, return:6061- Video/media URLs62- Title63- Description64- Tags6566## Examples6768### Example 1: Download Bilibili Video6970**User request**: "Download this Bilibili video: https://www.bilibili.com/video/BV1xx..."7172```731. createCrawlTask:74 - link: "https://www.bilibili.com/video/BV1xx..."752. Wait 30 seconds763. Poll getCrawlTaskStatus until success774. Return video URL, title, description, tags78```7980### Example 2: Download from Short Link8182**User request**: "下载这个视频 https://v.douyin.com/xxx"8384```851. createCrawlTask:86 - link: "https://v.douyin.com/xxx"872. Wait 30 seconds883. Poll getCrawlTaskStatus until success894. Return video URL, title, description, tags90```9192### Example 3: Extract for Re-publishing9394```951. createCrawlTask with source URL962. Poll getCrawlTaskStatus until completed973. Extract media URLs, title, description, tags984. Use content skill to save media995. Use publish skill to publish to target platforms100```101102## Task Status Values103104| Status | Description |105| ------- | ------------------------------ |106| pending | Task created, waiting to start |107| running | Task is actively processing |108| success | Task completed successfully |109| failed | Task failed with error |110111## Important Notes112113- Processing time varies by platform and content size114- Large videos may take longer to download115- Some platforms may have rate limits116- Media URLs are temporary - save to content library for permanent storage
yikart