# Reddit Data Extraction

> Reddit Data Extraction

- Skill: `lucadominguez/reddit-data-extraction` (Agent Skill)
- Install (CLI): `npx skillmds@latest add lucadominguez/reddit-data-extraction`
- Raw SKILL.md: https://api.skillmd.com/api/skills/lucadominguez/reddit-data-extraction/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: lucadominguez (https://skillmd.com/u/lucadominguez)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/lucadominguez/reddit-data-extraction

---

# Reddit Data Extraction

## Trigger

Use this skill when the user asks to:
- Download a user's posts or comment history from Reddit
- Archive subreddit content for research or training data
- Collect Reddit posts mentioning specific topics, compounds, or keywords
- Build a dataset from Reddit for AI fine-tuning

The user tracks research chemicals, nootropics, and niche pharmacology communities (r/NooTopics, r/Nootropics, r/DrugNerds, r/ResearchChemicals). Reddit experts like u/sirsadalot produce reference-quality content worth archiving.

## The Core Problem

**Reddit blocks all server-side requests.** Direct approaches return 403 HTML blocks, not JSON:
- `curl https://www.reddit.com/user/<name>/submitted.json` - blocked
- `curl https://old.reddit.com/user/<name>/submitted.json` - blocked
- Python `requests` with spoofed User-Agent - blocked
- Browser automation from VPS IPs - blocked or timed out

Do NOT waste time trying alternative User-Agent strings, old.reddit.com, or browser tools. They all fail from non-residential IPs.

## Solution: Arctic Shift API

**Primary endpoint:** `https://arctic-shift.photon-reddit.com/api/posts/search`

This is a public Reddit archive API. No authentication required. Rate limit: be polite (0.5-1s between requests).

### Fetching posts by author

```bash
curl -s "https://arctic-shift.photon-reddit.com/api/posts/search?author=<username>&limit=100&sort=desc"
```

Response structure:
- `data` array of Reddit post objects (full API schema)
- No explicit pagination metadata — you must use the `before` cursor

### Pagination pattern

The API returns max 100 posts per request. Paginate by passing `before=<created_utc>` of the last post in the current page:

```python
cursor = None
all_posts = []

while True:
    url = f'https://arctic-shift.photon-reddit.com/api/posts/search?author={username}&limit=100&sort=desc'
    if cursor:
        url += f'&before={cursor}'
    
    data = fetch_json(url)
    posts = data.get('data', [])
    
    if not posts:
        break
    
    all_posts.extend(posts)
    
    if len(posts) < 100:
        break  # partial page = last page
    
    cursor = posts[-1].get('created_utc')
```

### Key post fields for extraction

| Field | Description |
|---|---|
| `title` | Post title |
| `selftext` | Post body (plain text) |
| `selftext_html` | Post body (HTML) |
| `subreddit` | Subreddit name |
| `permalink` | Relative URL path |
| `created_utc` | Unix timestamp |
| `score` | Upvote score |
| `num_comments` | Comment count |
| `id` | Reddit post ID (e.g., `1ubz6li`) |
| `crosspost_parent` | Set if this is a crosspost (duplicate) |

### Pitfalls

1. **JSON control characters:** `selftext` fields contain raw newlines and control characters that break `json.loads()`. Save the curl response to a file first, then read with `json.load()` (file handles encoding). If using `json.loads()` on a string, pass `strict=False` or use the `json_parse` helper.

2. **Crosspost duplicates:** Crossposts from the user's own posts appear as separate entries with `crosspost_parent` set. Filter them out when counting unique content: `not post.get('crosspost_parent')`.

3. **Large payloads:** Responses with 100 posts can exceed 2MB. Don't pipe curl directly to Python — save to a temp file first to avoid truncation.

4. **Output file sizes:** A typical Reddit power user produces ~250 posts. The raw JSON dump will be 3-5MB. The cleaned JSONL for training will be ~800KB. Plan disk space accordingly.

## Output Formats for AI Training

When the goal is AI training data, produce THREE output formats:

### 1. Raw JSON dump (`all_posts_raw.json`)
Complete API response for reproducibility. Full Reddit schema including metadata, awards, flair, crosspost info.

### 2. Clean JSONL (`all_posts.jsonl`)
One JSON object per line. Keep only content-relevant fields:
```json
{"title": "...", "selftext": "...", "subreddit": "...", "score": 123, "num_comments": 45, "created_utc": 1782071336, "permalink": "/r/...", "id": "1ubz6li"}
```
This is the recommended format for most training pipelines (HuggingFace datasets, fine-tuning scripts, etc.).

### 3. Individual markdown files (`rundowns/`)
One `.md` per post for human browsing and reference. Good for long-form reference posts. Naming: `{index:03d}_{sanitized_title}.md`

## Classifying "Rundown" Posts

The user often wants specifically the "single standing rundowns" — long-form, original, reference-quality posts. Heuristic:
```python
is_rundown = (
    len(post.get('selftext', '') or '') > 500
    and not post.get('crosspost_parent')
)
```
Adjust the threshold based on the author's typical post length. For u/sirsadalot, 500 chars reliably separates long-form analyses from short discussion posts.

## Limitations

- **Comments not included:** This endpoint returns posts only. For comment history, explore Arctic Shift's comment search endpoint or use PRAW with authentication.
- **Recency:** Arctic Shift may lag behind live Reddit by hours to days depending on their ingestion pipeline.
- **Deleted/removed content:** Posts removed by the author or moderators may not appear, depending on when Arctic Shift last indexed them.

## Example: Full download session

See `references/sirsadalot-download-example.md` for the complete script used in a real download of 251 posts across 3 pages.