active-research

# Active Research Analyze any topic, domain, or paper and generate a beautiful HTML report using **Actionbook Browser** — featuring SPA-aware navigation, network idle detection, batch operations, and intelligent page analysis. ## Enhanced Browser Capabilities | Capability | Description | |------------|-------------| | Page load wait | `wait-idle` — monitors fetch/XHR until network settles | | SPA content | `wait-fn` — wait for JS conditions before extracting | | Page understanding | `snapshot --filter interactive --max-tokens N` — focused, budget-friendly | | Popups blocking | `--auto-dismiss-dialogs` — auto-handle alert/confirm/prompt | | Load speed | `--block-images` — skip images for faster text extraction | | Page stability | `--no-animations` — freeze CSS transitions | | Error detection | `console --level error` — check for page issues | | Multi-step forms | `batch` — execute multiple actions in one call | | Element debugging | `info <selector>` — inspect visibility, position, properties | | Change tracking | `snapshot --diff` — only see what changed | | Anti-detection | `--stealth` + `fingerprint rotate` for protected sites | | Auth management | `storage set` — inject JWT/tokens for gated content | | One-shot fetch | `browser fetch <url>` — navigate+wait+extract+close in one command | | Static page speed | `--lite` — HTTP-first, browser fallback only if needed | | Anti-scrape URLs | `--rewrite-urls` — x.com→xcancel.com, reddit→old.reddit | | Wait tuning | `--wait-hint` — domain-aware wait (fast/normal/slow/heavy) | | Log correlation | `--session-tag` — tag all operations for debugging | ## Usage ``` /active-research <topic> /active-research <topic> --output ./reports/my-report.json ``` Or simply tell Claude: "Research XXX and generate a report" ### Parameters | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `topic` | Yes | - | The subject to research (any text) | | `--output` | No | `./output/<topic-slug>.json` | Output path for JSON report | ### Topic Detection | Pattern | Type | Strategy | |---------|------|----------| | `arxiv:XXXX.XXXXX` | Paper | **arXiv Advanced Search** + ar5iv deep read | | `doi:10.XXX/...` | Paper | Resolve DOI, then **arXiv Advanced Search** for related work | | Academic keywords (paper, research, model, algorithm) | Academic topic | **arXiv Advanced Search** + Google for non-academic sources | | URL | Specific page | Fetch and analyze the page | | General text | Topic research | Google search + arXiv Advanced Search if relevant | ## Architecture ``` ┌──────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────┐ │ Claude │────▶│ Actionbook │────▶│ Web Pages │────▶│ Extract │ │ Code │ │ Browser │ │ (multiple) │ │ Content │ └──────────┘ └──────────────┘ └──────────────┘ └─────┬────┘ │ │ wait-idle │ │ SPA / dynamic │ │ │ │ batch ops │ │ protected │ │ │ │ --stealth │ │ mobile-only │ │ │ │ snapshot │ └───────────────┘ │ │ └──────────────┘ │ │ ┌──────────────┐ ┌──────────────┐ │ ├─────────▶│ Actionbook │ │ arXiv Adv. │ │ │ │ search/get │────▶│ Search Form │───────────▶│ │ │ (selectors) │ │ (40+ fields) │ │ │ └──────────────┘ └──────────────┘ │ │ │ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ │ │ Open in │◀────│ json-ui │◀────│ Write JSON │◀───────────┘ │ Browser │ │ render │ │ Report │ Synthesize └──────────┘ └──────────────┘ └──────────────┘ ``` ## MUST USE Actionbook CLI **Always use `actionbook browser` commands for web browsing. NEVER use any other method to access the web:** - **NEVER** use `curl`, `wget`, `httpie`, or any HTTP CLI tool via bash - **NEVER** use `python -c "import requests"` or any scripting-language HTTP library via bash - **NEVER** use WebFetch or WebSearch tools - **ONLY** use `actionbook browser` and `actionbook search`/`actionbook get` commands If you need web content, the PREFERRED path is: `actionbook browser fetch <url> --format text --json` (one-shot). For interactive multi-step workflows, use: `actionbook browser open <url>` → `actionbook browser wait-idle` → `actionbook browser text`. ## Browser Flags — Research Defaults **CRITICAL: Always use these flags when opening the browser for research.** ```bash # PREFERRED: One-shot fetch (I1) — handles open+wait+extract+close automatically actionbook --block-images --rewrite-urls browser fetch "<url>" --format text --json # For interactive multi-step workflows, use explicit open: actionbook --block-images --auto-dismiss-dialogs --no-animations --rewrite-urls browser open "<url>" ``` | Flag | Why | |------|-----| | `--block-images` | Skip image downloads — 2-5x faster page load for text extraction | | `--auto-dismiss-dialogs` | Prevent alert/confirm/prompt from blocking automation | | `--no-animations` | Freeze CSS animations — stable snapshots, no timing issues | | `--rewrite-urls` | Rewrite x.com→xcancel.com, reddit→old.reddit to avoid anti-bot blocking | | `--wait-hint <hint>` | Domain-aware wait: `instant`, `fast`, `normal`, `slow`, `heavy`, or ms | | `--session-tag <tag>` | Tag all operations for log correlation and debugging | | `--lite` (fetch only) | Try HTTP first, skip browser for static pages (Wikipedia, docs, blogs) | For sites with anti-bot protection, add `--stealth`: ```bash actionbook --block-images --auto-dismiss-dialogs --no-animations --stealth --rewrite-urls browser open "<url>" ``` ## Navigation Pattern — ALWAYS Follow **Option A: One-shot fetch (PREFERRED for read-only page extraction):** ```bash # Single command: navigate → wait (domain-aware) → extract → close actionbook --block-images --rewrite-urls browser fetch "<url>" --format text --json # For static pages (Wikipedia, docs, blogs), add --lite to skip browser entirely: actionbook --rewrite-urls browser fetch "<url>" --format text --lite --json # For accessibility tree: actionbook --block-images --rewrite-urls browser fetch "<url>" --format snapshot --max-tokens 2000 --json ``` **Option B: Interactive multi-step pattern (for forms, clicks, multi-page flows):** ```bash # Step 1: Navigate actionbook browser open "<url>" # or: goto, click a link # Step 2: Wait for load (MANDATORY in v2) actionbook browser wait-idle # Wait for fetch/XHR to settle # Step 3: Extract content actionbook browser text [selector] # Extract text # OR actionbook browser snapshot --filter interactive --max-tokens 500 # Understand page structure ``` **Why `wait-idle` is critical:** - SPAs (React, Vue, Next.js) load content via fetch/XHR after initial HTML - Without waiting, `text` returns empty or incomplete content - `wait-idle` monitors all pending network requests, waits until quiet for 500ms **For pages that load content dynamically after network settles:** ```bash actionbook browser wait-idle actionbook browser wait-fn "document.querySelector('.results')" # Wait for specific element actionbook browser text ".results" ``` ## Complete Workflow > **REMINDER:** Every web access in this workflow MUST use `actionbook browser` commands. > Using `curl`, `wget`, `python requests`, or any other HTTP tool is **strictly forbidden**. > The bash tool should ONLY be used for `actionbook` CLI commands and local file operations (json-ui render, `open`). ### Step 1: Plan Search Strategy Based on the topic, generate 5-8 search queries from different angles: - Core definition / overview - Latest developments / news - Technical details / implementation - Comparisons / alternatives - Expert opinions / analysis - Use cases / applications **Search order — ALWAYS query Actionbook API first, then search:** | Step | Action | Why | |------|--------|-----| | **Step 2 (FIRST)** | **Query Actionbook API** | Get verified selectors for arXiv, ar5iv, and other known sites BEFORE browsing. | | **Step 3 (SECOND)** | **arXiv Advanced Search** | Use Actionbook selectors for multi-field, filtered academic search. | | **Step 4 (THIRD)** | Google / Bing search | Supplement with blogs, news, code, discussions, non-academic sources. | ### Step 2: Query Actionbook API for Selectors (ALWAYS DO THIS FIRST) **BEFORE browsing any URL, query Actionbook's indexed selectors.** ```bash # Search for indexed actions by domain actionbook search "<keywords>" -d "<domain>" # Get detailed selectors for a specific page actionbook get "<domain>:/<path>:<area>" ``` **Pre-indexed sites useful for research:** | Site | area_id | Key Selectors | |------|---------|---------------| | arXiv Advanced Search | `arxiv.org:/search/advanced:default` | **40+ selectors**: field select, term input, category checkboxes, date range filters | | ar5iv paper | `ar5iv.labs.arxiv.org:/html/{paper_id}:default` | `h1.ltx_title_document`, `div.ltx_authors`, `div.ltx_abstract`, `section.ltx_section` | | Google Scholar | `scholar.google.com:/:default` | `#gs_hdr_tsi` (search), `#gs_hdr_tsb` (submit) | | arXiv homepage | `arxiv.org:/:default` | Global search across 2.4M+ articles | **For any URL you plan to visit**, run `actionbook search "<keywords>" -d "<domain>"` to check if it's indexed. ### Step 3: arXiv Search (URL-First, Form as Backup) **LESSON LEARNED: arXiv form submission via browser automation is unreliable. Use URL-based search as the PRIMARY method.** **Option A: URL-based search (PRIMARY — most reliable):** ```bash # Simple keyword search actionbook --block-images --auto-dismiss-dialogs --no-animations browser open "https://arxiv.org/search/?query=large+language+model+agent&searchtype=all" actionbook browser wait-idle actionbook browser text "#main-container" # Advanced URL search with filters # searchtype: all, title, author, abstract # start: result offset (0, 50, 100, ...) actionbook browser open "https://arxiv.org/search/?query=Rust+machine+learning&searchtype=all&start=0" actionbook browser wait-idle actionbook browser text "#main-container" ``` **Search strategy: Start broad, then narrow:** 1. First search: broad terms (e.g., `"Rust" "machine learning"`) — aim for 50+ results 2. If too few results (< 10): broaden further, remove date/category filters 3. If too many results (> 200): add more specific terms, use `searchtype=title` 4. Try 2-3 different query angles (e.g., framework names, use cases, benchmarks) **Option B: Form interaction via batch (BACKUP — use if URL search is insufficient):** ```bash # Open arXiv with research flags actionbook --block-images --auto-dismiss-dialogs --no-animations browser open "https://arxiv.org/search/advanced" actionbook browser wait-idle # Use batch for form — fewer round-trips, more reliable cat <<'EOF' | actionbook browser batch --delay 150 { "actions": [ {"kind": "click", "selector": "#terms-0-field"}, {"kind": "click", "selector": "option[value='title']"}, {"kind": "type", "selector": "#terms-0-term", "text": "large language model agent"}, {"kind": "click", "selector": "#classification-computer_science"}, {"kind": "click", "selector": "#date-filter_by-3"}, {"kind": "type", "selector": "#date-from_date", "text": "2025-01-01"}, {"kind": "type", "selector": "#date-to_date", "text": "2026-02-23"}, {"kind": "click", "selector": "button:has-text('Search'):nth(2)"} ], "stopOnError": true } EOF actionbook browser wait-idle actionbook browser text "#main-container" # If batch form submission fails (page shows form again instead of results): # → Fall back to Option A URL-based search immediately # → Do NOT retry the form — it wastes time ``` **arXiv search capabilities (from indexed selectors — for Option B):** | Capability | Selector | |------------|----------| | Search field (Title/Author/Abstract) | `#terms-0-field` select | | Search term | `#terms-0-term` input | | Add boolean terms | `button "Add another term +"` | | Filter: Computer Science | `#classification-computer_science` | | Filter: Physics, Math, etc. | `#classification-physics`, `#classification-mathematics` | | Date: past 12 months | `#date-filter_by-1` radio | | Date: specific year | `#date-filter_by-2` radio + `#date-year` | | Date: custom range | `#date-filter_by-3` radio + `#date-from_date` / `#date-to_date` |