Reddit Data Extraction

Reddit Data Extraction

Trigger

Use this skill when the user asks to:

• Download a user's posts or comment history from Reddit
• Archive subreddit content for research or training data
• Collect Reddit posts mentioning specific topics, compounds, or keywords
• Build a dataset from Reddit for AI fine-tuning

The user tracks research chemicals, nootropics, and niche pharmacology communities (r/NooTopics, r/Nootropics, r/DrugNerds, r/ResearchChemicals). Reddit experts like u/sirsadalot produce reference-quality content worth archiving.

The Core Problem

Reddit blocks all server-side requests. Direct approaches return 403 HTML blocks, not JSON:

• curl https://www.reddit.com/user/<name>/submitted.json - blocked
• curl https://old.reddit.com/user/<name>/submitted.json - blocked
• Python requests with spoofed User-Agent - blocked
• Browser automation from VPS IPs - blocked or timed out

Do NOT waste time trying alternative User-Agent strings, old.reddit.com, or browser tools. They all fail from non-residential IPs.

Solution: Arctic Shift API

Primary endpoint: https://arctic-shift.photon-reddit.com/api/posts/search

This is a public Reddit archive API. No authentication required. Rate limit: be polite (0.5-1s between requests).

Fetching posts by author

curl -s "https://arctic-shift.photon-reddit.com/api/posts/search?author=<username>&limit=100&sort=desc"

Response structure:

• data array of Reddit post objects (full API schema)
• No explicit pagination metadata — you must use the before cursor

Pagination pattern

The API returns max 100 posts per request. Paginate by passing before=<created_utc> of the last post in the current page:

cursor = None
all_posts = []

while True:
    url = f'https://arctic-shift.photon-reddit.com/api/posts/search?author={username}&limit=100&sort=desc'
    if cursor:
        url += f'&before={cursor}'
    
    data = fetch_json(url)
    posts = data.get('data', [])
    
    if not posts:
        break
    
    all_posts.extend(posts)
    
    if len(posts) < 100:
        break  # partial page = last page
    
    cursor = posts[-1].get('created_utc')

Key post fields for extraction

Pitfalls

1. JSON control characters: selftext fields contain raw newlines and control characters that break json.loads(). Save the curl response to a file first, then read with json.load() (file handles encoding). If using json.loads() on a string, pass strict=False or use the json_parse helper.

2. Crosspost duplicates: Crossposts from the user's own posts appear as separate entries with crosspost_parent set. Filter them out when counting unique content: not post.get('crosspost_parent').

3. Large payloads: Responses with 100 posts can exceed 2MB. Don't pipe curl directly to Python — save to a temp file first to avoid truncation.

4. Output file sizes: A typical Reddit power user produces ~250 posts. The raw JSON dump will be 3-5MB. The cleaned JSONL for training will be ~800KB. Plan disk space accordingly.

Output Formats for AI Training

When the goal is AI training data, produce THREE output formats:

1. Raw JSON dump (all_posts_raw.json)

Complete API response for reproducibility. Full Reddit schema including metadata, awards, flair, crosspost info.

2. Clean JSONL (all_posts.jsonl)

One JSON object per line. Keep only content-relevant fields:

{"title": "...", "selftext": "...", "subreddit": "...", "score": 123, "num_comments": 45, "created_utc": 1782071336, "permalink": "/r/...", "id": "1ubz6li"}

This is the recommended format for most training pipelines (HuggingFace datasets, fine-tuning scripts, etc.).

3. Individual markdown files (rundowns/)

One .md per post for human browsing and reference. Good for long-form reference posts. Naming: {index:03d}_{sanitized_title}.md

Classifying "Rundown" Posts

The user often wants specifically the "single standing rundowns" — long-form, original, reference-quality posts. Heuristic:

is_rundown = (
    len(post.get('selftext', '') or '') > 500
    and not post.get('crosspost_parent')
)

Adjust the threshold based on the author's typical post length. For u/sirsadalot, 500 chars reliably separates long-form analyses from short discussion posts.

Limitations

• Comments not included: This endpoint returns posts only. For comment history, explore Arctic Shift's comment search endpoint or use PRAW with authentication.
• Recency: Arctic Shift may lag behind live Reddit by hours to days depending on their ingestion pipeline.
• Deleted/removed content: Posts removed by the author or moderators may not appear, depending on when Arctic Shift last indexed them.

Example: Full download session

See references/sirsadalot-download-example.md for the complete script used in a real download of 251 posts across 3 pages.