# Threads-Research-Scraper-Skill

> Conduct live topic, keyword, account, and comment research & web scraping on Meta Threads (threads.net) using Playwright headless browser automation, post page navigation, and reply thread extraction.

- Skill: `karlyu130/threads-research-scraper-skill` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add karlyu130/threads-research-scraper-skill`
- Raw SKILL.md: https://api.skillmd.com/api/skills/karlyu130/threads-research-scraper-skill/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: KarlYu130 (https://skillmd.com/u/karlyu130)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/karlyu130/threads-research-scraper-skill

---


# Threads-Research-Scraper-Skill

This skill provides an end-to-end automated workflow to scrape live posts, extract child reply/comment threads, analyze user engagement, and synthesize research reports on **Meta Threads (`threads.net`)**.

---

## 1. When to Use
Use this skill whenever you need to:
- Scrape or extract live Threads post data AND child comments/replies for a topic, keyword, or hashtag (e.g. `#HKUST`).
- Research posts, discussions, or trends on Meta Threads (`threads.net`).
- Analyze user sentiment, debate threads, or community feedback on Threads.
- Sort search results chronologically (newest first) or by algorithmic relevance.

---

## 2. Environment Prerequisites

Ensure Playwright and Chromium browser binaries are installed:

```bash
pip install playwright
playwright install chromium
```

---

## 3. Workflow & Tooling

```
┌─────────────────────────┐     1. Search Query & Scroll       ┌─────────────────────────┐
│                         │ ─────────────────────────────────> │                         │
│  Playwright Scraper     │                                    │  Chromium Engine        │
│  (scripts/scraper.py)   │ <───────────────────────────────── │  (Executes JavaScript)  │
│                         │     2. Extract Post URLs           └────────────┬────────────┘
└────────────┬────────────┘                                                 │
             │                                                              │ 3. Deep Post Page Navigation
             │ 4. Extract Main Post + Child Comments                        ▼
             └─────────────────────────────────────────────────── ┌─────────────────────────┐
                                                                  │  threads.net/@user/post │
                                                                  └─────────────────────────┘
```

### Execution Steps:
1. **Search & Link Harvesting**: Executes `python scripts/threads_live_scraper.py --query <terms>`, loads `https://www.threads.net/search?q=<query>`, and collects post URLs (`a[href*="/post/"]`).
2. **Deep Post Navigation**: Navigates into each post URL directly.
3. **Post & Comment Extraction**: Captures the `main_post` content and all child `comments` / user replies beneath it.
4. **Report Generation**: Parses extracted JSON data into structured markdown research artifacts.

---

## 4. Script Usage

```bash
# Scrape posts and child comment/reply threads
python scripts/threads_live_scraper.py --query "HKUST" "香港科技大學" --output threads_posts_and_comments.json
```

### Output JSON Structure (`threads_posts_and_comments.json`):
```json
{
  "HKUST": [
    {
      "url": "https://www.threads.net/@user/post/DZ3fKhijjvK",
      "main_post": "Main post content...",
      "comments": [
        "First reply comment...",
        "Second reply comment..."
      ]
    }
  ]
}
```