marketing-reddit

name

marketing-reddit

description

Pull structured data from Reddit (posts, threads, comments, and question clusters) through a headless browser that bypasses Reddit's JS bot-challenge. Use when Reddit's public .json API is 403-blocked from a server/cloud IP, or when mining a subreddit for content gaps, questions, or post data. Triggers on "mine reddit", "reddit-miner", "scrape subreddit", "reddit questions", "reddit 403", "reddit blocked", "pull reddit posts/comments".

marketing-reddit

Reddit hard-blocks the public .json API for datacenter/cloud IPs and serves a JS bot-challenge to anything automated. The decisive signal is the HeadlessChrome token in the User-Agent — override it to a clean Chrome/... UA and headless Chrome passes the challenge. The same-origin in-page fetch() then carries the clearance cookie and returns real JSON.

This tool drives agent-browser (headless) with a clean UA — optionally through a residential proxy — clears the challenge once per run, then fetches .json via in-page fetch(), metering real wire bytes (proxy bandwidth) and cost. Question detection + clustering are ported in-tool (no Python dependency).

Written in TypeScript, run with bun (runs .ts directly, zero install).

Why headless works here

Signal	Headless default	What we send	Verdict
User-Agent	…HeadlessChrome/…	…Chrome/… (clean)	decisive — clean UA passes
navigator.webdriver	true	true (unchanged)	not enough alone to block

--proxy/--user-agent apply at agent-browser daemon launch, so the tool runs agent-browser close --all at the start of a run to guarantee they take effect (this also closes other agent-browser sessions).

Setup

• bun installed (https://bun.sh) and agent-browser (npm i -g agent-browser).
• Proxy is optional. From a clean/residential IP it works direct. From a flagged datacenter/cloud IP (e.g. a CI box) you need a residential proxy.

# Store a proxy in machine-local config (chmod 600, OUTSIDE the skill repo — repo is public)
bun ~/.claude/skills/marketing-reddit/cli.ts setup --proxy "http://user:pass@host:port"

# Verify everything needed to run
bun ~/.claude/skills/marketing-reddit/cli.ts doctor

Proxy resolution order: --proxy flag → REDDIT_PROXY env → ~/.config/reddit-miner/config.json. The proxy secret is never read from or written to the skill repo. Use --no-proxy to force direct.

Setup flow — guide the user through this

When the user first asks to set this up, don't dump manual steps — drive it. Run doctor, then fix each FAIL yourself (ask before any global install or before handling a proxy secret). <cli> below is bun <this-skill-dir>/cli.ts.

1. Run <cli> doctor first. It reports each requirement as PASS/FAIL with the exact fix. Work down the failures in order.

2. bun runtime FAIL → bun isn't installed. Offer to run:

   curl -fsSL https://bun.sh/install | bash
   

3. agent-browser installed FAIL → offer to run:

   npm i -g agent-browser
   

4. browser engine (Chrome) reachable FAIL → agent-browser has no browser to drive. Offer to run (downloads a Chromium it controls; existing Chrome/Brave is auto-detected too):

   agent-browser install
   

5. proxy credential resolved is INFO, not a failure. Ask the user: do they have a residential/ISP proxy? Reddit blocks datacenter IPs, so a cloud/CI box needs one; a clean home IP often works direct.

   • If yes, take the proxy URL and store it for them — never paste it into the repo:

     <cli> setup --proxy "http://user:pass@host:port"
     

   • If no, tell them it'll run direct and only works from a clean IP. They can add a proxy later with setup.

6. Re-run <cli> doctor until it prints RESULT: READY. Don't proceed past a FAIL — a failing engine or a flagged IP means every mine returns nothing.

7. Smoke test once READY:

   <cli> mine --subreddit <their-subreddit> --threads 5
   

   Confirm it returns clustered questions and a small bandwidth/cost line.

For convenience, suggest the user alias the CLI: alias reddit-miner='bun <this-skill-dir>/cli.ts'.

Usage

# Busiest posts (title, selftext, score, comments, flair, author)
bun ~/.claude/skills/marketing-reddit/cli.ts posts --subreddit Retatrutide --sort top --time month --limit 30

# A single thread's full comment tree (raw Reddit JSON)
bun ~/.claude/skills/marketing-reddit/cli.ts thread --url https://www.reddit.com/r/Retatrutide/comments/<id>/<slug>/

# Mine questions from POSTS + COMMENTS, mark answered/unanswered, cluster the gaps
bun ~/.claude/skills/marketing-reddit/cli.ts mine --subreddit Retatrutide --time month --threads 30 \
  --topic-keywords "reta|retatrutide|dose|side effect"

--sort: top|hot|new|rising|controversial (--time applies to top/controversial). --keep-open: leave the browser session running for inspection. --no-proxy: force a direct connection even if a proxy is configured. --no-rotate: keep the proxy's stored sessid instead of minting a fresh one (see below).

Sticky-session auto-rotation + exit-IP failover

A sticky exit IP is required so the clearance cookie stays valid for every .json fetch in the run. Two vendor shapes:

Vendor	Sticky shape	What we mint
Oxylabs	sessid-… in the URL/userinfo	fresh sessid-<id> per run
DataImpulse sticky (gw.dataimpulse.com:1000X, user login__cr.us)	;sessid.<id> on the username	inject/refresh ;sessid.<id> per run

Without a sessid, DataImpulse pins one IP per sticky port — a flagged exit then fails every morning until someone hand-rotates. The tool now:

1. Mints a fresh sessid at process start (stable within the run).
2. On challenge fail, rotates immediately — new sessid → new residential IP → kill the agent-browser daemon (so the new --proxy takes effect) → retry. Default 5 exit IPs (REDDIT_MAX_EXIT_IPS); 2 page loads per exit.

Stored config is never mutated. --no-rotate keeps the configured URL as-is (no mint at start; clear still rotates on fail if the vendor shape is known).

Concurrent callers

The tool drives one agent-browser daemon per session name (default reddit-miner) and kills its own session's daemon whenever the proxy rotates. Two callers sharing the default session interfere: the second caller's --proxy/--user-agent are silently ignored ("daemon already running"), and each caller's daemon kill takes down the other's in-flight challenge run — which then looks exactly like a flagged proxy pool. When callers can overlap (parallel cron loops), give each its own session: REDDIT_MINER_SESSION=<caller-name>. Serializing callers with a lock works too, but per-session isolation is the fix.

Output

• posts → {subreddit, sort, posts: [...], bandwidth}
• thread → {thread: <raw Reddit listing JSON>, bandwidth}
• mine → {threads_scanned, posts_considered, questions, n_questions, n_unanswered, clusters_unanswered, top_questions, bandwidth}
  • questions[].source is reddit (comment) or reddit-post (selftext) — "more than just comments".

Every run also prints a one-line bandwidth/cost summary to stderr.

Bandwidth & cost

bandwidth reports real wire bytes (transferSize, post-gzip, incl. headers) — what a metered proxy bills. Set REDDIT_PROXY_USD_PER_GB (default 8) for cost estimates. Reference: a 30-thread mine ≈ 1 MB wire ≈ $0.008 at $8/GB (~955 runs per GB).

Failure modes

• could not clear Reddit JS challenge after N exit IP(s) → the tool already rotated through REDDIT_MAX_EXIT_IPS sticky sessions and every one was flagged (or the vendor shape is unknown so it couldn't rotate). Check the proxy balance / country filter, or bump a dead DataImpulse sticky port (:10001→:10002 — ports die independently of sessid rotation).
• fetchJson failed … http 403 / blocked → challenge expired mid-run; the tool retries the fetch. A fresh process re-clears (and re-mints a sessid).