Reddit Fetch
Reddit's public JSON API works by appending .json to any Reddit URL — but Reddit now hard-blocks automated access. curl 403s essentially every time (host AND container, regardless of User-Agent), and even a cold Playwright navigation to reddit.com hits a "You've been blocked by network security" challenge page. The reliable method is to arrive at Reddit through a DuckDuckGo result redirect: that sets a Reddit session cookie which unlocks direct .json access for the rest of the browser session.
Primary method: the DuckDuckGo-hop unlock
Step 1 - the DDG hop. Do this once per session before any .json fetch.
mcp__playwright__browser_navigate to https://html.duckduckgo.com/html/?q=site:reddit.com/r/SUBREDDIT+YOUR+QUERY
- Grab the first result's full href - it's a DDG redirect that includes a
rut token (https://duckduckgo.com/l/?uddg=...&rut=...). The token is required; navigating to the bare /l/?uddg= without it 400s.() => document.querySelector('.result__a')?.href
browser_navigate to that full redirect href. It lands on a real www.reddit.com page (title = the post/subreddit title, not "Blocked") and sets the session cookie. The result doesn't have to be the exact thread you want - landing on any real Reddit page sets the cookie.
If Playwright errors with Browser is already in use, a stale instance is holding the profile - pkill -f ms-playwright-mcp (or the profile dir named in the error) and retry.
Step 2 - direct .json now works. For the rest of the session, navigate Playwright straight to any .json URL and JSON.parse(document.body.innerText). Use www.reddit.com (not old.reddit.com) for browser navigation. Full recency sorting (sort=new&t=week) is available.
browser_navigate to e.g. https://www.reddit.com/r/SUBREDDIT/search.json?q=QUERY&restrict_sr=on&sort=new&t=week&limit=25
browser_evaluate, always wrapped in try/catch (return document.body.innerText.slice(0,200) on failure so you can see a challenge page if the session lapsed - just re-do the hop):() => {
try {
const data = JSON.parse(document.body.innerText);
return data.data.children.map(c => ({
t: c.data.title, s: c.data.score, n: c.data.num_comments, id: c.data.id
}));
} catch (e) { return document.body.innerText.slice(0, 200); }
}
- For a thread, navigate to
.../comments/POST_ID.json?limit=30&sort=top and parse data[0] (post) and data[1].data.children (comments).
JSON shapes (same for browser and curl)
# Listing - swap hot for new/top/rising; for top add &t=day|week|month|year|all
/r/SUBREDDIT/hot.json?limit=15
# Post + comments - JSON array where [0]=post, [1]=comment tree
/r/SUBREDDIT/comments/POST_ID.json?limit=20
# Search within a subreddit
/r/SUBREDDIT/search.json?q=QUERY&restrict_sr=on&sort=new&limit=15
- Listings:
.data.children[].data has title, score, num_comments, author, id.
- Threads:
[0].data.children[0].data is the post; [1].data.children[] (filter kind == "t1") are comments with author, score, body, and nested replies of the same shape.
- Truncate long comment bodies (e.g.
body.slice(0, 300) in JS, .body[:300] in jq 1.7+) to keep output readable.
Fallbacks
- More comments per thread: load the rendered thread page (after the hop) and scrape the
shreddit DOM - it returns more comments than .json?limit=:() => ({
title: document.querySelector('shreddit-post')?.getAttribute('post-title'),
comments: [...document.querySelectorAll('shreddit-comment')].map(c => ({
author: c.getAttribute('author'),
score: c.getAttribute('score'),
text: c.querySelector('.md')?.innerText
}))
})
- No Playwright at all: use Claude for Chrome to open the thread /
.json URL and read it off the page.
- curl (last resort, expect 403): direct curl with a browser User-Agent used to work and is faster when it does, but it now gets 403'd essentially always - changing the UA doesn't help. Only worth a single quick try if you're already shelling out and a browser isn't available; on 403, go straight to the DDG hop.
UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
curl -s -L -o /tmp/reddit_result.txt -w "%{http_code}" -H "User-Agent: $UA" \
'https://old.reddit.com/r/SUBREDDIT/hot.json?limit=15'
jq -r '.data.children[] | .data | "\(.title)\n \(.score) pts | \(.num_comments) comments | u/\(.author) | id: \(.id)\n"' /tmp/reddit_result.txt
Fetch to a temp file (-o) then parse; -w "%{http_code}" prints the status; -L follows redirects; single-quote the URL so the shell doesn't eat &.
Rate limiting
Reddit rate-limits aggressively even once you're unblocked:
- Don't fire parallel requests - run them sequentially with
sleep 2/sleep 3 (or brief pauses between navigations). Fetch one listing, parse it, then fetch threads one at a time.
- Empty response (0 bytes): wait 3-5s and retry. HTTP 429: back off 10-15s. A challenge page mid-session means the cookie lapsed - re-do the DDG hop.
Source: ykdojo/claude-code-tips → skills/reddit-fetch/SKILL.md
1---
2name: reddit-fetch
3description: "Fetch content from Reddit via its JSON API using a browser session (DuckDuckGo-hop unlock). Use when accessing Reddit URLs, researching topics on Reddit, or when Reddit returns 403/blocked errors."
4category: backend-and-data
5source_repo: ykdojo/claude-code-tips
6source_path: "skills/reddit-fetch/SKILL.md"
7source_url: https://github.com/ykdojo/claude-code-tips/blob/HEAD/skills/reddit-fetch/SKILL.md
8metadata:
9 source: https://skillmd.com/skills/ykdojo/reddit-fetch
10---
11
12
13# Reddit Fetch
14
15Reddit's public JSON API works by appending `.json` to any Reddit URL — but Reddit now hard-blocks automated access. **curl 403s essentially every time (host AND container, regardless of User-Agent), and even a cold Playwright navigation to `reddit.com` hits a `"You've been blocked by network security"` challenge page.** The reliable method is to arrive at Reddit *through a DuckDuckGo result redirect*: that sets a Reddit session cookie which unlocks direct `.json` access for the rest of the browser session.
16
17## Primary method: the DuckDuckGo-hop unlock
18
19**Step 1 - the DDG hop.** Do this once per session before any `.json` fetch.
20
211. `mcp__playwright__browser_navigate` to `https://html.duckduckgo.com/html/?q=site:reddit.com/r/SUBREDDIT+YOUR+QUERY`
222. Grab the first result's **full href** - it's a DDG redirect that includes a `rut` token (`https://duckduckgo.com/l/?uddg=...&rut=...`). The token is required; navigating to the bare `/l/?uddg=` without it 400s.
23 ```js
24 () => document.querySelector('.result__a')?.href
25 ```
263. `browser_navigate` to that full redirect href. It lands on a real `www.reddit.com` page (title = the post/subreddit title, **not** "Blocked") and sets the session cookie. The result doesn't have to be the exact thread you want - landing on *any* real Reddit page sets the cookie.
27
28If Playwright errors with `Browser is already in use`, a stale instance is holding the profile - `pkill -f ms-playwright-mcp` (or the profile dir named in the error) and retry.
29
30**Step 2 - direct `.json` now works.** For the rest of the session, navigate Playwright straight to any `.json` URL and `JSON.parse(document.body.innerText)`. Use `www.reddit.com` (not `old.reddit.com`) for browser navigation. Full recency sorting (`sort=new&t=week`) is available.
31
321. `browser_navigate` to e.g. `https://www.reddit.com/r/SUBREDDIT/search.json?q=QUERY&restrict_sr=on&sort=new&t=week&limit=25`
332. `browser_evaluate`, **always wrapped in try/catch** (return `document.body.innerText.slice(0,200)` on failure so you can see a challenge page if the session lapsed - just re-do the hop):
34 ```js
35 () => {
36 try {
37 const data = JSON.parse(document.body.innerText);
38 return data.data.children.map(c => ({
39 t: c.data.title, s: c.data.score, n: c.data.num_comments, id: c.data.id
40 }));
41 } catch (e) { return document.body.innerText.slice(0, 200); }
42 }
43 ```
443. For a thread, navigate to `.../comments/POST_ID.json?limit=30&sort=top` and parse `data[0]` (post) and `data[1].data.children` (comments).
45
46## JSON shapes (same for browser and curl)
47
48```text
49# Listing - swap hot for new/top/rising; for top add &t=day|week|month|year|all
50/r/SUBREDDIT/hot.json?limit=15
51# Post + comments - JSON array where [0]=post, [1]=comment tree
52/r/SUBREDDIT/comments/POST_ID.json?limit=20
53# Search within a subreddit
54/r/SUBREDDIT/search.json?q=QUERY&restrict_sr=on&sort=new&limit=15
55```
56
57- Listings: `.data.children[].data` has `title`, `score`, `num_comments`, `author`, `id`.
58- Threads: `[0].data.children[0].data` is the post; `[1].data.children[]` (filter `kind == "t1"`) are comments with `author`, `score`, `body`, and nested `replies` of the same shape.
59- Truncate long comment bodies (e.g. `body.slice(0, 300)` in JS, `.body[:300]` in jq 1.7+) to keep output readable.
60
61## Fallbacks
62
63- **More comments per thread:** load the rendered thread page (after the hop) and scrape the `shreddit` DOM - it returns more comments than `.json?limit=`:
64 ```js
65 () => ({
66 title: document.querySelector('shreddit-post')?.getAttribute('post-title'),
67 comments: [...document.querySelectorAll('shreddit-comment')].map(c => ({
68 author: c.getAttribute('author'),
69 score: c.getAttribute('score'),
70 text: c.querySelector('.md')?.innerText
71 }))
72 })
73 ```
74- **No Playwright at all:** use Claude for Chrome to open the thread / `.json` URL and read it off the page.
75- **curl (last resort, expect 403):** direct curl with a browser User-Agent used to work and is faster when it does, but it now gets 403'd essentially always - changing the UA doesn't help. Only worth a single quick try if you're already shelling out and a browser isn't available; on 403, go straight to the DDG hop.
76 ```bash
77 UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
78 curl -s -L -o /tmp/reddit_result.txt -w "%{http_code}" -H "User-Agent: $UA" \
79 'https://old.reddit.com/r/SUBREDDIT/hot.json?limit=15'
80 jq -r '.data.children[] | .data | "\(.title)\n \(.score) pts | \(.num_comments) comments | u/\(.author) | id: \(.id)\n"' /tmp/reddit_result.txt
81 ```
82 Fetch to a temp file (`-o`) then parse; `-w "%{http_code}"` prints the status; `-L` follows redirects; single-quote the URL so the shell doesn't eat `&`.
83
84## Rate limiting
85
86Reddit rate-limits aggressively even once you're unblocked:
87
88- **Don't fire parallel requests** - run them sequentially with `sleep 2`/`sleep 3` (or brief pauses between navigations). Fetch one listing, parse it, then fetch threads one at a time.
89- Empty response (0 bytes): wait 3-5s and retry. HTTP 429: back off 10-15s. A challenge page mid-session means the cookie lapsed - re-do the DDG hop.
90
91---
92
93**Source:** [`ykdojo/claude-code-tips`](https://github.com/ykdojo/claude-code-tips) → `skills/reddit-fetch/SKILL.md`