deep-research

# Deep Research Drive `openbrowser-ai` to investigate a topic across multiple web sources and produce a cited markdown report plus structured JSON. Two modes: - **flat synthesis** (default) -- decompose query into 3-7 sub-questions, dispatch one parallel sub-agent per sub-question (each owns one tab), merge into one cited report. - **drilldown** (auto-detected from prompt phrasing: "deep dive", "exhaustive", "recursive", "drilldown", "thorough") -- same as flat, plus a second wave of parallel sub-agents on findings flagged `needs_depth=true`. Hard cap depth=2, max 3 follow-up sub-agents per parent. Output paths (relative to current project root): - `local_docs/research/YYYY-MM-DD-<slug>.md` - `local_docs/research/YYYY-MM-DD-<slug>.json` **Architecture (mandatory):** the orchestrating Claude session (the one running this skill) MUST dispatch parallel sub-agents via `/dispatching-parallel-agents`, one sub-agent per sub-question. Each sub-agent owns exactly ONE tab. Sub-agents do not open additional tabs. The orchestrator merges per-agent findings into one report. Why one tab per sub-agent and not `asyncio.gather` over tabs in a single `-c` call: a single Python coroutine driving N tabs through one daemon serializes navigation events at the CDP layer, contends for the LLM-extraction worker, and cannot make independent decisions about pagination or follow-up clicks per tab. Dispatching real Claude sub-agents (each with its own context window and its own browser tab) gives true parallelism, independent reasoning per tab, and isolates failures so one bad page doesn't poison the rest. Hard rules: - One sub-agent = one tab. Sub-agents must NOT call `navigate(url, new_tab=True)` to spawn additional tabs. - All sub-agents share the same daemon (and so the same Chrome process). Tabs are isolated; navigation in one tab does not affect another. - Each sub-agent writes its findings to its own JSON file under `local_docs/research/_partial/<slug>-NN.json`. The orchestrator reads and merges these. - The orchestrator never drives tabs itself. It only plans, dispatches, merges, renders, verifies, cleans up. If a first-wave sub-agent returns <2 findings, the orchestrator dispatches a Step 2b retry sub-agent with broader search strategy (alternative engines, query reformulation, lower thresholds). Still `-c`-only: the skill never calls `openbrowser-ai -p`. Variables persist across `-c` calls in the daemon namespace. **Session reuse:** Step 0 checks `openbrowser-ai daemon status`. If a daemon is already running (warm browser), the skill reuses it and operates in NEW tabs (never disturbs the user's existing tabs). If no daemon, the skill auto-starts one on first `-c` call. Every factual claim in the report carries a footnote citation `[N]`. Verifier fails the run if uncited prose is found. ## Setup Verify install: ```bash openbrowser-ai --help ``` Install if missing: ```bash # macOS / Linux curl -fsSL https://openbrowser.me/install.sh | sh # Windows PowerShell irm https://openbrowser.me/install.ps1 | iex ``` No LLM API key required. The skill drives the daemon via `openbrowser-ai -c` only, which executes raw CDP / JS through the daemon's Python namespace and never invokes a model. (The `-p` "prompt mode" of the CLI is a separate code path that loads `get_llm()` and requires an OpenAI / Anthropic / Google key per `cli.py:434-490`. This skill explicitly avoids `-p`.) Set the headless env var so the daemon starts without a visible browser window (the default in `daemon/server.py` is already `headless: True`, but a user config file can override it; this env var wins over config): ```bash export OPENBROWSER_HEADLESS=true ``` Prepare output dir at the project root (NOT user home): ```bash mkdir -p local_docs/research ``` ## Workflow ### Step 0 -- Session check Enforce headless mode and detect whether a daemon is already running. If yes, reuse it (operate in NEW tabs only). If no, the next `-c` call auto-starts one. `OPENBROWSER_HEADLESS=true` is set here so the daemon spawned by the first `-c` call inherits it, even if the user's config file sets `headless: false`. Already-running daemons are unaffected (their browser was opened at start time). ```bash export OPENBROWSER_HEADLESS=true if openbrowser-ai daemon status 2>&1 | grep -qi 'running\|listening\|pid'; then echo "Reusing existing daemon -- will work in new tabs" export DEEP_RESEARCH_REUSED=1 else echo "No daemon running -- will start fresh headless session" export DEEP_RESEARCH_REUSED=0 fi ``` Snapshot existing tabs so cleanup leaves them untouched: ```bash openbrowser-ai -c - <<'EOF' state = await browser.get_browser_state_summary() _preexisting_tab_ids = {t.target_id for t in state.tabs} if state.tabs else set() print(f"Pre-existing tabs: {len(_preexisting_tab_ids)}") EOF ``` ### Step 1 -- Plan Decompose the user query into sub-questions and pick the mode. Daemon namespace persists `_plan` across later `-c` calls. ```bash openbrowser-ai -c - <<'EOF' import json, re, datetime, os QUERY = """<USER_QUERY>""" # paste exact user query here # Daemon CWD often != shell CWD. Hard-code the absolute project root. # Set this to the shell CWD at the start of the run; do NOT rely on os.getcwd(). PROJECT_ROOT = "<ABSOLUTE_PATH_TO_PROJECT_ROOT>" # e.g. /Users/foo/myproject # Auto-detect mode DRILL_RE = re.compile(r"\b(deep ?dive|exhaustive|recursive|drill ?down|thorough)\b", re.I) mode = "drilldown" if DRILL_RE.search(QUERY) else "flat" # Slug = first 60 chars, lowercase, non-alnum -> '-', collapse repeats def slugify(s): s = re.sub(r"[^a-z0-9]+", "-", s.lower())[:60] return s.strip("-") or "research" today = datetime.date.today().isoformat() slug = slugify(QUERY) research_dir = os.path.join(PROJECT_ROOT, "local_docs", "research") os.makedirs(research_dir, exist_ok=True) base = os.path.join(research_dir, f"{today}-{slug}") md_path, json_path = f"{base}.md", f"{base}.json" # Bump suffix if collision n = 2 while os.path.exists(md_path): md_path, json_path = f"{base}-{n}.md", f"{base}-{n}.json" n += 1 # Decompose: 3-7 sub-questions. Keep tight, non-overlapping, each answerable from web. # This is a heuristic split; replace with your own decomposition for the actual query. sub_questions = [ # e.g. "What is X?", # "Who are the main actors / vendors / authors?", # "What are recent (last 12 months) developments?", # "What are the trade-offs / criticisms?", # "What concrete numbers / benchmarks exist?", ] assert 3 <= len(sub_questions) <= 7, "need 3-7 sub-questions" _plan = { "query": QUERY, "mode": mode, "generated_at": datetime.datetime.now().astimezone().isoformat(), "sub_questions": sub_questions, "md_path": md_path, "json_path": json_path, } print(json.dumps(_plan, indent=2)) EOF ``` Edit the `QUERY`, `PROJECT_ROOT`, and `sub_questions` list before running. `PROJECT_ROOT` MUST be an absolute path: the daemon runs in its own working directory (usually wherever the daemon was first started), so relative paths land in the wrong place. Use the shell `pwd` output as the value. Verify output looks right before continuing. ### Step 2a -- Dispatch parallel sub-agents (one tab per agent) The orchestrator (the Claude session running this skill) MUST invoke `/dispatching-parallel-agents` and dispatch one sub-agent per sub-question. All sub-agents share the same `openbrowser-ai` daemon. Each sub-agent owns exactly one tab, the one it opens at the start of its run. **Hard rules for the orchestrator:** - Send ONE message containing N parallel `Agent` tool calls, where N = `len(_plan["sub_questions"])`. - Each sub-agent gets the prompt template below, parameterized with: `SUB_QUESTION`, `AGENT_INDEX` (zero-padded 2 digits, used in output filename), `PROJECT_ROOT` (absolute path). - After dispatch, wait for all sub-agents to return. Do not begin Step 4 until every partial JSON file under `local_docs/research/_partial/` is on disk. - The orchestrator does NOT drive any tab in this step. **Sub-agent prompt template** (copy into each `Agent` tool call's `prompt` argument): ``` You are a deep-research sub-agent. Your job: investigate ONE sub-question in ONE Chrome tab and write findings to a JSON file. Sub-question: <SUB_QUESTION> Agent index: <AGENT_INDEX> Project root: <PROJECT_ROOT> Hard constraints: - You own ONE tab. The tab is the one you open at the start of this task. - NEVER pass new_tab=True to navigate(). Reuse your one tab for every page. - Do not switch to other tabs. - Do not call openbrowser-ai daemon stop. The orchestrator owns daemon lifecycle. - Visit at least 3 result URLs from at least 3 different domains. - For each URL, extract one finding with: claim (one sentence), exact url, supporting quote (verbatim, max 40 words), domain, confidence (low|medium|high), needs_depth (bool, true if a deeper follow-up would meaningfully sharpen the claim). - Return at least 2 findings; 3 is ideal. - Write the findings JSON array to: <PROJECT_ROOT>/local_docs/research/_partial/agent-<AGENT_INDEX>.json Workflow (run via openbrowser-ai -c - heredocs, all in this one bash session): 1. Open exactly one tab on Google search: openbrowser-ai -c - <<'EOF' await navigate("https://www.google.com/search?q=<URL_ENCODED_SUB_QUESTION>", new_tab=True) await wait(2) state = await browser.get_browser_state_summary() tab_id = state.tabs[-1].target_id _MY_TAB = tab_id[-4:] print(f"my tab: {_MY_TAB}") EOF 2. Scrape the SERP for top 5 result URLs. 3. For each of the top 3 results, navigate IN THE SAME TAB (no new_tab=True), wait 2s, capture body innerText, pick the sentence with the most word-overlap vs the sub-question (40-280 chars, >3-letter keyword overlap >= 1). 4. Write the findings JSON array to the partial file. 5. Print a one-line summary: "agent <AGENT_INDEX>: <N> findings written". Return: a one-line confirmation that the partial JSON file was written, and its absolute path. Do not include the findings in your text reply -- the orchestrator will read them from disk. ``` The orchestrator's pre-step (run once before dispatching agents): ```bash openbrowser-ai -c - <<'EOF' import os PROJECT_ROOT = "<ABSOLUTE_PATH_TO_PROJECT_ROOT>" partial_dir = os.path.join(PROJECT_ROOT, "local_docs", "research", "_partial") os.makedirs(partial_dir, exist_ok=True) # Wipe stale partials from prior runs of the same query for f in os.listdir(partial_dir): if f.startswith("agent-") and f.endswith(".json"): os.remove(os.path.join(partial_dir, f)) print(f"partial dir ready: {partial_dir}") EOF ``` After all sub-agents return, the orchestrator collects findings: ```bash openbrowser-ai -c - <<'EOF' import os, json global _findings PROJECT_ROOT = "<ABSOLUTE_PATH_TO_PROJECT_ROOT>" partial_dir = os.path.join(PROJECT_ROOT, "local_docs", "research", "_partial") _findings = [] for fname in sorted(os.listdir(partial_dir)): if not (fname.startswith("agent-") and fname.endswith(".json")): continue path = os.path.join(partial_dir, fname) try: with open(path) as fh: arr = json.load(fh) if isinstance(arr, list): # normalize exact_url -> url (sub-agents may use either key) for item in arr: if "exact_url" in item and "url" not in item: item["url"] = item.pop("exact_url") _findings.extend(arr) except Exception as e: print(f"skip {path}: {e}") print(f"merged {len(_findings)} findings from {len(os.listdir(partial_dir))} partials") EOF ``` If any sub-question's partial file is missing or has <2 findings, fall through to Step 2b for that one. ### Step 2b -- Retry weak sub-questions with broader sub-agents If a sub-question came back with <2 findings, do NOT use `openbrowser-ai -p`. The `-p` mode requires an LLM API key (OpenAI / Anthropic / Google) because it runs the full Browser Agent loop with LLM-driven navigation, see `cli.py:434-490` `get_llm()`. This skill is `-c`-only by design: the daemon executes raw CDP / JS and needs no API key. Identify weak sub-questions, then dispatch a fresh round of parallel sub-agents (same `/dispatching-parallel-agents` pattern as Step 2a) with a broader prompt. The retry sub-agents reuse the same single-tab discipline and write to `agent-retry-NN.json` partials. ```bash openbrowser-ai -c - <<'EOF' from collections import Counter global _retry_subqs counts = Counter(f.get("sub_question") for f in _findings) _retry_subqs = [sq for sq in _plan["sub_questions"] if counts.get(sq, 0) < 2] print(f"Retry targets: {len(_retry_subqs)}") for sq in _retry_subqs: print(f" - {sq}") EOF ``` The orchestrator then dispatches one parallel sub-agent per weak sub-question with this retry prompt template (same single-tab rule, broader search strategy): ``` You are a deep-research RETRY sub-agent. Your job: investigate ONE sub-question that the first-wave sub-agent could not satisfy. Use ONE Chrome tab and write findings to a JSON file. Sub-question: <SUB_QUESTION> Agent index: <AGENT_INDEX> (filename agent-retry-<AGENT_INDEX>.json) Project root: <PROJECT_ROOT> Hard constraints: - You own ONE tab. NEVER pass new_tab=True to navigate() after the first call. - Use openbrowser-ai -c only. NEVER use openbrowser-ai -p (it requires an LLM API key and we explicitly avoid that path). - The first-wave attempt failed: heuristic sentence picker returned <2 findings. This means the page text either had no high-overlap sentences for the question, or the SERP returned thin sources. Try one or more of: 1. Reformulate the search query (try 2-3 alternative phrasings, pick the one with the best SERP). 2. Search a different engine: try Bing or DuckDuckGo if Google was thin. URLs: https://www.bing.com/search?q=... or https://duckduckgo.com/?q=... 3. Lower the sentence-length floor for the heuristic (e.g. 30 chars instead of 40), or accept partial-match sentences with overlap >= 1 word. 4. For factual sub-questions ("when was X released"), check Wikipedia directly: https://en.wikipedia.org/wiki/Special:Search?search=... - Same JSON shape as Step 2a sub-agents. - Write to: <PROJECT_ROOT>/local_docs/research/_partial/agent-retry-<AGENT_INDEX>.json - Return at least 2 findings. If still <2 after the broader strategy, write whatever you got (even 0-1) and surface the limitation in your reply. ``` After all retry sub-agents return, merge their partials into `_findings`: ```bash openbrowser-ai -c - <<'EOF' import os, json global _findings PROJECT_ROOT = "<ABSOLUTE_PATH_TO_PROJECT_ROOT>"