Threads-Research-Scraper-Skill

Threads-Research-Scraper-Skill

This skill provides an end-to-end automated workflow to scrape live posts, extract child reply/comment threads, analyze user engagement, and synthesize research reports on Meta Threads (threads.net).

1. When to Use

Use this skill whenever you need to:

• Scrape or extract live Threads post data AND child comments/replies for a topic, keyword, or hashtag (e.g. #HKUST).
• Research posts, discussions, or trends on Meta Threads (threads.net).
• Analyze user sentiment, debate threads, or community feedback on Threads.
• Sort search results chronologically (newest first) or by algorithmic relevance.

2. Environment Prerequisites

Ensure Playwright and Chromium browser binaries are installed:

pip install playwright
playwright install chromium

3. Workflow & Tooling

┌─────────────────────────┐     1. Search Query & Scroll       ┌─────────────────────────┐
│                         │ ─────────────────────────────────> │                         │
│  Playwright Scraper     │                                    │  Chromium Engine        │
│  (scripts/scraper.py)   │ <───────────────────────────────── │  (Executes JavaScript)  │
│                         │     2. Extract Post URLs           └────────────┬────────────┘
└────────────┬────────────┘                                                 │
             │                                                              │ 3. Deep Post Page Navigation
             │ 4. Extract Main Post + Child Comments                        ▼
             └─────────────────────────────────────────────────── ┌─────────────────────────┐
                                                                  │  threads.net/@user/post │
                                                                  └─────────────────────────┘

Execution Steps:

1. Search & Link Harvesting: Executes python scripts/threads_live_scraper.py --query <terms>, loads https://www.threads.net/search?q=<query>, and collects post URLs (a[href*="/post/"]).
2. Deep Post Navigation: Navigates into each post URL directly.
3. Post & Comment Extraction: Captures the main_post content and all child comments / user replies beneath it.
4. Report Generation: Parses extracted JSON data into structured markdown research artifacts.

4. Script Usage

# Scrape posts and child comment/reply threads
python scripts/threads_live_scraper.py --query "HKUST" "香港科技大學" --output threads_posts_and_comments.json

Output JSON Structure (threads_posts_and_comments.json):

{
  "HKUST": [
    {
      "url": "https://www.threads.net/@user/post/DZ3fKhijjvK",
      "main_post": "Main post content...",
      "comments": [
        "First reply comment...",
        "Second reply comment..."
      ]
    }
  ]
}