| name | deep-research |
| description | Generic parallel multi-source deep-research orchestrator. Fans out across 8 data sources in parallel (X via monid/tikhub, Reddit via monid, Hacker News via Algolia, GitHub repos + issues, Polymarket Gamma, YouTube with transcripts via monid, Exa neural web search via monid) and dumps quotable evidence as JSON + human-readable markdown in a single command. Use whenever the user says "do deep research on X", "what's the discourse on Y", "research this topic", "give me a primer on Z", asks for cross-source validation of a thesis, or needs broad evidence before drafting any long-form piece (briefing, post, report, memo). |
| license | MIT |
| compatibility | Requires python3, monid CLI, MONID_API_KEY (funded monid balance). Optional MONID_BIN for non-PATH installs. |
Deep Research — parallel multi-source orchestrator
🟢 What this is. A single Python script that fans out across 8 data sources in parallel and dumps quotable evidence into a research folder. Use it whenever you need cross-source convergence on a topic before writing anything substantive.
🔴 What this is NOT. It does not synthesize a brief, an opinion, or a post. The output is raw evidence for the writer to read and synthesize. Voice and structure happen downstream — never inside the orchestrator.
When to use
| Triggering moment | Why this skill |
|---|
| User pastes a tweet/URL and says "research this topic" or "do a deep dive on this" | Run on the topic before manually pulling individual sources. Saves 5-10 sequential calls. |
| User asks "what's the discourse on X" / "what are people saying about Y" | This is literally what the tool does. |
| Pre-flight before any long-form artifact (deep post, briefing, founder profile, product comparison, ecosystem map) | The 8-source sweep finds links and quotes that one-source spelunking would miss. |
| Need to validate a thesis with cross-source convergence | If Reddit + HN + X all surface the same pattern, that's a real market signal worth a thesis. |
🟡 Not the right tool for: single-event news posts where the source is one tweet (just fetch that tweet directly), or re-research of topics already covered.
How to invoke
cd skills/deep-research/scripts
python3 research.py "your topic in quotes" [flags]
Common flag combinations:
python3 research.py "agent harness engineering"
python3 research.py "Stripe Sessions 2026 announcements" --depth=quick
python3 research.py "openai vs anthropic enterprise 2026" --depth=deep --yt-transcripts=4
python3 research.py "Mitchell Hashimoto" --sources=x,github_repos,github_issues,hn
python3 research.py "GPT-5.5 launch" --days=7
python3 research.py "claude code skills" --out=/path/to/research/claude-code-skills/
Output
🟢 Default output: ./research/<slug>/research-<date>.{json,md} (relative to current working directory).
🟢 Override with --out=<dir> for project-specific paths.
| File | What's in it | Use it for |
|---|
research-<date>.md | Human-readable cluster, sorted by engagement | Read first to triage which threads are worth pulling |
research-<date>.json | Full structured evidence with all fields | Pull verbatim quotes, engagement numbers, URLs into the downstream artifact |
The markdown groups results by source (X, Reddit, HN, GitHub repos, GitHub issues, Polymarket, YouTube, Exa) and within each section sorts by engagement signal. Each entry has the URL, engagement counts, author, date, and a quoted excerpt.
The 8 sources, what they give you
| Source | Cost | Auth | What it's good for |
|---|
🟢 X / Twitter (via monid tikhub/twitter/web/fetch_search_timeline) | ~$0.0015/page | MONID_API_KEY | Verbatim tweets with real engagement (likes/RT/replies/views), ranked by engagement. ~20 tweets/page; deep depth pages + a Latest sweep. The single most valuable source for op-ed work. |
🟢 Reddit (via monid Apify trudax/reddit-scraper-lite) | ~$0.05-0.10/run | MONID_API_KEY | Operator/user opinion threads with top comments. Anti-corporate angle. |
🟢 Hacker News (via monid exa/contents → Algolia JSON) | ~$0.0022/call | MONID_API_KEY | Developer-community discussion. Great for tooling/product launches. |
🟢 GitHub repos (via monid exa/contents → api.github.com) | ~$0.0022/call | MONID_API_KEY | Star counts, release dates, license, related projects. Critical for ecosystem maps. |
🟢 GitHub issues/PRs (via monid exa/contents → api.github.com) | ~$0.0022/call | MONID_API_KEY | Active development threads, contested features, recent bugs. Ground-truth project status. |
🟢 Polymarket (via monid exa/contents → Gamma search) | ~$0.0022/call | MONID_API_KEY | Money-weighted predictions. Odds on regulatory deadlines, election outcomes, product launches. |
| 🟢 YouTube (via monid Apify: search + transcript) | ~$0.03 / 2 transcripts | MONID_API_KEY | Recent videos + full transcripts for the top N. Often game-changing for talk/interview-driven topics. |
🟢 Exa neural web search (via monid blockrun.ai/api/v1/exa/search) | ~$0.011/call | MONID_API_KEY | Recent grounded web content + research papers. Supports a category filter (research paper, news, github, tweet, …). Returns inline text + highlights. |
🟢 Total per-run cost: ~$0.10-0.20 at default depth. No monthly subscription.
🟢 EVERY source routes through monid — one auth (MONID_API_KEY), one balance, zero per-vendor keys. Paid sources (X, Reddit, YouTube, Exa) use native monid endpoints; sources with no native monid endpoint (HN Algolia, GitHub REST, Polymarket Gamma) are proxied through blockrun.ai/api/v1/exa/contents, which returns the upstream JSON body verbatim (raw bytes, no markdown cleaning) for ~$0.0022/call.
🔴 Do NOT proxy JSON through blockrun.ai/api/v1/surf/web/fetch — it runs a markdown cleaner that collapses whitespace inside JSON strings and mangles embedded URLs, producing invalid JSON on large/prose-heavy responses (GitHub issue bodies, Polymarket descriptions). Use exa/contents (raw passthrough) instead — that's what _monid.fetch_json does.
🟡 Per-source budget rules:
- Reddit costs scale with results × 0.0057 + a base scrape fee. A "deep" run can hit $0.25.
- YouTube transcripts cost $0.0075 per video — start with
--yt-transcripts=2, raise to 4+ only if the topic is YouTube-heavy.
- X (tikhub) is
$0.0015/page; default depth pulls 2 pages ($0.003), deep pulls 3 + a Latest sweep.
- HN/GitHub/Polymarket proxy calls are ~$0.0022 each. Polymarket Gamma is capped at
limit_per_type=8 — exa livecrawl 504s on the ~1MB payload above that.
Example workflow
1. User: "do a deep dive on Greg Isenberg's agent agency podcast"
2. Read the source URL (the tweet, the YouTube link)
3. Run: python3 research.py "Greg Isenberg agent agency Hermes Orgo"
4. Wait ~60-90s for all 8 sources
5. Open research-<date>.md, scan the X + Reddit + Exa sections
6. For each high-signal post, pull the verbatim text from research-<date>.json
7. Draft your artifact with verbatim quotes lifted directly from the JSON
🔴 Critical rule: this skill produces research input, not draft output. Never copy/paste sentences from research-<date>.md into the artifact you're writing. Always pull the underlying primary source (tweet text, blog excerpt, GitHub description) from the .json and synthesize in your own voice downstream.
Voice contract
This skill must not synthesize editorial prose or take positions. Its only job is to retrieve, normalize, and rank.
If the script ever starts emitting things like "the takeaway is...", "this represents a shift...", or "the bottom line:", that's a bug — remove it. The synthesis is what humans (or downstream editorial workflows) do.
Files
| Path | What |
|---|
SKILL.md | This file |
scripts/research.py | The orchestrator (CLI entry) |
scripts/sources/_monid.py | Shared monid runner: run_monid (retry/backoff) + fetch_json (raw-JSON via exa/contents) — used by ALL sources |
scripts/sources/reddit.py | Reddit via monid Apify |
scripts/sources/hackernews.py | HN Algolia via monid exa/contents |
scripts/sources/polymarket.py | Polymarket Gamma via monid exa/contents |
scripts/sources/github.py | GitHub REST via monid exa/contents |
scripts/sources/x_twitter.py | X via monid tikhub |
scripts/sources/youtube.py | YouTube via monid Apify |
scripts/sources/exa.py | Exa neural search via monid (blockrun.ai proxy) |
Required env
| Env var | Used by | Required for |
|---|
MONID_API_KEY | _monid.py → ALL sources | Everything — every source routes through monid |
🟢 One key for everything. Set MONID_API_KEY in your environment and ensure your monid balance is funded.
🟡 Optional env var: MONID_BIN — path to the monid CLI binary. Defaults to a monid lookup on $PATH. Set this if you've installed monid in a non-standard location.
🟡 Missing-auth behavior: if MONID_API_KEY is unset, sources return empty and the orchestrator logs warnings but still completes (with no data). Ensure the key + a funded monid balance before running.
Installing monid
This skill depends on the monid CLI. Install it with:
npm install -g monid
npm install monid
export MONID_BIN="$(pwd)/node_modules/.bin/monid"
Get an API key at monid.dev and export it:
export MONID_API_KEY=your_key_here
Known gotchas
| Gotcha | Mitigation |
|---|
🔴 Never proxy JSON through surf/web/fetch — its markdown cleaner corrupts large/prose-heavy JSON (mangles embedded URLs → invalid JSON) | Use exa/contents (raw passthrough). _monid.fetch_json already does this. Verified clean on 166KB GitHub issue payloads. |
🔴 exa/contents 504s (CRAWL_LIVECRAWL_TIMEOUT) on ~1MB+ responses, and transiently even on small ones | fetch_json retries the no-results case with backoff (2 retries). Polymarket Gamma is capped at limit_per_type=8; keep GitHub per_page<=15. |
🟡 Reddit's Apify upVotes field returns 1 as default when the scraper skips engagement metadata | Module filters on "any score or any comments or has body or has top comments" instead of a hard floor. Trust the human reading the markdown to rank by relevance. |
| 🟡 Reddit can return 0 results for low-volume topics | Try a broader query, then filter manually. |
🔴 GitHub /search/issues requires is:issue or is:pr (returns 422 otherwise as of 2026-05) | Already handled in github.py — module appends is:issue automatically. |
🟡 HN: web/fetch-style cleaners mangle Algolia story_text/_highlightResult (legacy note; exa/contents avoids this) | hackernews.py still requests attributesToHighlight=["none"] + a tight field whitelist to keep payloads small and URL-free — defensive even with the raw path. |
🔴 tikhub (X) intermittently returns an HTML error page (Unexpected token '<') under rate-limiting — ~1 in 3 calls | Already handled: _monid.py detects transient HTML/UNKNOWN errors and retries with exponential backoff (2 retries default). If X still returns 0, re-run or lower depth. |
🟡 tikhub (X) has no date param — date filtering is client-side from each tweet's created_at | x_twitter.py parses created_at and filters to the --days window; undated tweets are kept rather than dropped. Use short keywords (1-2 words), not sentences. |
| 🟡 Polymarket Gamma keyword search ignores very short terms (e.g. "AI") | Use longer queries ("artificial intelligence"); the orchestrator passes the full topic. |
🔴 YouTube scraper input field is searchQueries, NOT searchKeywords | Already handled in youtube.py. Apify schema change in Apr 2026. |
| 🟢 No persistent state. Each run is independent. | Re-running on the same topic costs the same as the first run — sources are not cached. Plan accordingly. |
Updating this skill
When a source's input schema changes, GitHub adds a new search filter, or a monid endpoint changes — update the relevant sources/*.py and bump the Known gotchas section. Everything goes through sources/_monid.py:
- Native monid endpoint (X, Reddit, YouTube, Exa): call
run_monid(provider, endpoint, body=/query=/path=, retries=) and normalize result["output"].
- No native endpoint (HN, GitHub, Polymarket — arbitrary JSON REST): call
fetch_json(url) which proxies through exa/contents and returns parsed JSON. NEVER use surf/web/fetch for JSON (markdown cleaner corrupts it).
To add a source: monid discover → monid inspect the schema → write a thin sources/<name>.py using whichever helper fits. Treat this SKILL.md as a living spec.
Credits
Originally inspired by mvanhorn/last30days-skill. The 30-day default window came from that upstream; everything else (the 8-source fan-out, the monid routing, the _monid.py helpers, the per-source modules) was rebuilt from scratch.
