reddit-fetch

# Reddit Fetch Reddit's public JSON API works by appending `.json` to any Reddit URL — but Reddit now hard-blocks automated access. **curl 403s essentially every time (host AND container, regardless of User-Agent), and even a cold Playwright navigation to `reddit.com` hits a `"You've been blocked by network security"` challenge page.** The reliable method is to arrive at Reddit *through a DuckDuckGo result redirect*: that sets a Reddit session cookie which unlocks direct `.json` access for the rest of the browser session. ## Primary method: the DuckDuckGo-hop unlock **Step 1 - the DDG hop.** Do this once per session before any `.json` fetch. 1. `mcp__playwright__browser_navigate` to `https://html.duckduckgo.com/html/?q=site:reddit.com/r/SUBREDDIT+YOUR+QUERY` 2. Grab the first result's **full href** - it's a DDG redirect that includes a `rut` token (`https://duckduckgo.com/l/?uddg=...&rut=...`). The token is required; navigating to the bare `/l/?uddg=` without it 400s. ```js () => document.querySelector('.result__a')?.href ``` 3. `browser_navigate` to that full redirect href. It lands on a real `www.reddit.com` page (title = the post/subreddit title, **not** "Blocked") and sets the session cookie. The result doesn't have to be the exact thread you want - landing on *any* real Reddit page sets the cookie. If Playwright errors with `Browser is already in use`, a stale instance is holding the profile - `pkill -f ms-playwright-mcp` (or the profile dir named in the error) and retry. **Step 2 - direct `.json` now works.** For the rest of the session, navigate Playwright straight to any `.json` URL and `JSON.parse(document.body.innerText)`. Use `www.reddit.com` (not `old.reddit.com`) for browser navigation. Full recency sorting (`sort=new&t=week`) is available. 1. `browser_navigate` to e.g. `https://www.reddit.com/r/SUBREDDIT/search.json?q=QUERY&restrict_sr=on&sort=new&t=week&limit=25` 2. `browser_evaluate`, **always wrapped in try/catch** (return `document.body.innerText.slice(0,200)` on failure so you can see a challenge page if the session lapsed - just re-do the hop): ```js () => { try { const data = JSON.parse(document.body.innerText); return data.data.children.map(c => ({ t: c.data.title, s: c.data.score, n: c.data.num_comments, id: c.data.id })); } catch (e) { return document.body.innerText.slice(0, 200); } } ``` 3. For a thread, navigate to `.../comments/POST_ID.json?limit=30&sort=top` and parse `data[0]` (post) and `data[1].data.children` (comments). ## JSON shapes (same for browser and curl) ```text # Listing - swap hot for new/top/rising; for top add &t=day|week|month|year|all /r/SUBREDDIT/hot.json?limit=15 # Post + comments - JSON array where [0]=post, [1]=comment tree /r/SUBREDDIT/comments/POST_ID.json?limit=20 # Search within a subreddit /r/SUBREDDIT/search.json?q=QUERY&restrict_sr=on&sort=new&limit=15 ``` - Listings: `.data.children[].data` has `title`, `score`, `num_comments`, `author`, `id`. - Threads: `[0].data.children[0].data` is the post; `[1].data.children[]` (filter `kind == "t1"`) are comments with `author`, `score`, `body`, and nested `replies` of the same shape. - Truncate long comment bodies (e.g. `body.slice(0, 300)` in JS, `.body[:300]` in jq 1.7+) to keep output readable. ## Fallbacks - **More comments per thread:** load the rendered thread page (after the hop) and scrape the `shreddit` DOM - it returns more comments than `.json?limit=`: ```js () => ({ title: document.querySelector('shreddit-post')?.getAttribute('post-title'), comments: [...document.querySelectorAll('shreddit-comment')].map(c => ({ author: c.getAttribute('author'), score: c.getAttribute('score'), text: c.querySelector('.md')?.innerText })) }) ``` - **No Playwright at all:** use Claude for Chrome to open the thread / `.json` URL and read it off the page. - **curl (last resort, expect 403):** direct curl with a browser User-Agent used to work and is faster when it does, but it now gets 403'd essentially always - changing the UA doesn't help. Only worth a single quick try if you're already shelling out and a browser isn't available; on 403, go straight to the DDG hop. ```bash UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" curl -s -L -o /tmp/reddit_result.txt -w "%{http_code}" -H "User-Agent: $UA" \ 'https://old.reddit.com/r/SUBREDDIT/hot.json?limit=15' jq -r '.data.children[] | .data | "\(.title)\n \(.score) pts | \(.num_comments) comments | u/\(.author) | id: \(.id)\n"' /tmp/reddit_result.txt ``` Fetch to a temp file (`-o`) then parse; `-w "%{http_code}"` prints the status; `-L` follows redirects; single-quote the URL so the shell doesn't eat `&`. ## Rate limiting Reddit rate-limits aggressively even once you're unblocked: - **Don't fire parallel requests** - run them sequentially with `sleep 2`/`sleep 3` (or brief pauses between navigations). Fetch one listing, parse it, then fetch threads one at a time. - Empty response (0 bytes): wait 3-5s and retry. HTTP 429: back off 10-15s. A challenge page mid-session means the cookie lapsed - re-do the DDG hop.