| name | research-fetch |
| description | Highest-accuracy web page extraction for research. Parallel three-way extraction (Trafilatura + Readability.js + VLM full-page screenshot) reconciled by an LLM. Use when you need accurate, structured content from any URL — especially login-walled, JS-heavy, or bot-blocked sites. Complements chrome-devtools skill (which is for interactive UI automation). |
| metadata | {"openclaw":{"emoji":"🔬"}} |
Skill: research-fetch
Core idea: three extractors in parallel, then a VLM reconciles them against the full-page screenshot.
Highest-accuracy web content extraction, optimized for "correctness first, speed/cost second" research scenarios. Replaces one-off curl, web_fetch, or interactive CDP scraping when you need the page content as structured markdown + metadata.
When to Use What
| Task | Tool | Why |
|---|
| "Read this URL and summarize" | research-fetch | Parallel 3-way extraction → most accurate |
| Build a dataset from 100 URLs | research-fetch --no-vlm | Fast mode, ~3s/URL |
| Click buttons, fill forms, E2E tests | chrome-devtools | Interactive UI automation |
| Static HTML docs, RSS feeds | web_fetch / feedparser | Faster, no browser needed |
| Dashboards, tabular data on a login-walled site | research-fetch + --schema=... | Structured extraction |
| Screenshot-only (visual inspection) | chrome-devtools | Built-in |
| Long JS SPA that never settles | research-fetch | Waits for networkidle + scrolls for lazy-load |
Golden rule: chrome-devtools is for acting on a page; research-fetch is for reading a page accurately.
Architecture
URL
↓
Playwright (attaches to existing Chrome via CDP, or launches headless)
├─ Navigate, wait for domcontentloaded + networkidle
├─ Scroll to bottom (trigger lazy-load)
└─ Full-page JPEG screenshot
↓
3 parallel extractors:
A. Trafilatura 2.x (HTML → markdown, academic F1 > 0.94)
B. Readability.js (Mozilla Reader Mode, injected in-page)
C. Full-page screenshot → VLM (Claude Sonnet 4.6 or similar)
↓
LLM reconciler (same VLM, 16k output tokens):
- Use Readability HTML as primary body (it keeps structure)
- Cross-validate against Trafilatura for completeness
- Verify against the screenshot for "what's visible"
- Extract title/author/date/language/excerpt
- Compute a confidence score (0-1) based on 3-way agreement
↓
json-repair fallback (fixes common LLM JSON mistakes like unescaped quotes)
↓
Guard: if LLM markdown < 80% of Trafilatura → fall back to Trafilatura
Quick Start
1. One-time setup
cd path/to/skills/research-fetch
bash setup.sh
This creates .venv/ and installs: trafilatura, playwright, openai, json-repair, html2text, lxml_html_clean.
2. Run
bash run.sh https://example.com
bash run.sh https://example.com --no-vlm
bash run.sh https://example.com --no-cdp
bash run.sh https://example.com --md-only
bash run.sh https://example.com --viewport-only
3. Configure
All via environment variables:
| Env | Default | What |
|---|
RF_VLM_MODEL | claude-sonnet-4-6 | Any vision-capable chat model |
LITELLM_BASE_URL | http://localhost:4000/v1 | OpenAI-compatible endpoint |
LITELLM_KEY | sk-litellm-local | API key |
RF_CDP_URL | http://127.0.0.1:9222 | Chrome CDP endpoint (optional) |
Point LITELLM_BASE_URL at OpenAI, Anthropic, a LiteLLM gateway, or any other OpenAI-compatible server.
Output Schema
{
"url": "https://original.url/",
"final_url": "https://after-redirect/",
"title": "...",
"author": "...",
"published_at": "YYYY-MM-DD or ISO-8601",
"language": "zh | en | ja | ...",
"markdown": "Full main-content markdown (headings, lists, tables, code blocks, images, quotes preserved)",
"excerpt": "150-char summary",
"confidence": 0.95,
"rejected_noise": ["nav bar", "sidebar ads", ...],
"notable_media": [{"type": "image|video|table|code", "description": "..."}],
"links_outbound": ["github.com", "arxiv.org", ...],
"sources": {
"trafilatura_chars": 4830,
"readability_chars": 4721,
"html_chars": 189407,
"screenshot_kb": 1019
},
"tokens_used": {"in": 4698, "out": 1338},
"elapsed_s": 47.8,
"_raw_dir": "/path/to/raw_<timestamp>/"
}
Real-World Accuracy (2026-05-09)
| Page | Elapsed | Markdown chars | Confidence | Notes |
|---|
| example.com | 12.7s | 200 | 0.99 | Baseline |
| English tech blog (Simon Willison) | 47.8s | 4524 | 0.98 | Full structure preserved |
| Chinese tech article behind paywall (CSDN) | 54.1s | 804 | 0.72 | LLM honestly flagged truncation — doesn't hallucinate |
| 404 page | 13.2s | 84 | 0.85 | Correctly identified |
| MSN weather, Shenzhen | ~17s | structured JSON | — | 15+ fields: temp/feels/humidity/wind/pressure/UV/dew/AQI/alerts/17-day forecast |
Comparison vs. Alternatives
| Metric | curl | web_fetch | chrome-devtools | Firecrawl API | research-fetch |
|---|
| Runs JS / renders SPA | ❌ | ❌ | ✅ | ✅ | ✅ |
| Reuses logged-in Chrome | manual cookies | ❌ | ✅ | ❌ | ✅ (via CDP attach) |
| Main-content extraction | raw HTML | basic | single-source | single-source | 3-source reconciled |
| Honest paywall detection | shows teaser only | — | — | — | labels truncation |
| Confidence score | ❌ | ❌ | ❌ | ❌ | ✅ |
| Structured output | ❌ | basic | manual | LLM-friendly | markdown + metadata |
| Cost | free | free | free | paid | ~$0.01/URL |
| Throughput | highest | high | low (interactive) | high | low (research-grade) |
Bottom line: use research-fetch when accuracy matters more than cost or speed. For bulk ingest, use --no-vlm mode or fall back to Trafilatura alone.
Limitations
- Single task at a time (file lock on
/tmp/.cdp_scrape_lock), since the attached Chrome can't be used concurrently.
- Paywalled / login-walled content: returns only the visible portion; LLM explicitly labels the truncation rather than hallucinating.
- Infinite-scroll feeds (Weibo, Xiaohongshu timelines): captures only the first few viewports. Use a dedicated feed skill for those.
- Video bodies: only reads the thumbnail/page metadata, not video content.
- PDFs embedded in iframes: not currently supported. Use a PDF-specific skill.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|
playwright install chromium fails | Proxy blocks CDN | Run setup.sh with proxy env set, or playwright install --with-deps chromium |
| CDP attach returns 0 contexts | Chrome not running with --remote-debugging-port=9222 | Start Chrome with the flag, or use --no-cdp to launch headless |
LLM returns JSON but confidence is null | Parser repaired JSON but some fields missing | Check ~/.openclaw/workspace/data/tmp/research-fetch/_llm_debug/*_raw_response.txt |
| markdown much shorter than Trafilatura | LLM output hit max_tokens | Bump max_tokens in fetch.py (currently 16k) or switch to a larger-output model |
curl localhost:9222 times out | Local proxy intercepting localhost | unset http_proxy https_proxy before calling |
| 404 page "confidence 0.99" | Real 404 detected | Verify the URL or follow redirects manually |
File Layout
skills/research-fetch/
├── SKILL.md # this file
├── fetch.py # main script (~400 lines)
├── run.sh # bash wrapper with env sanitation
├── setup.sh # one-time venv + playwright install
├── requirements.txt # pinned Python deps
├── Readability.js # Mozilla's Reader Mode (bundled, 91 KB)
└── tests/
└── run_all.sh # regression tests across scenario categories
Generated artifacts go to ~/.openclaw/workspace/data/tmp/research-fetch/ by default (override with RF_OUT_DIR).
Integration Recipes
Use from Python
import subprocess, json
res = subprocess.run(
["bash", "skills/research-fetch/run.sh", url],
capture_output=True, text=True, timeout=120,
)
data = json.loads(res.stdout)
print(data["title"], data["confidence"])
Use from an agent prompt
When the user asks you to read/summarize a URL, don't browse interactively.
Instead shell out to:
bash ~/.openclaw/workspace/skills/research-fetch/run.sh <URL>
The JSON on stdout already contains title, author, date, confidence, and the
full markdown body. Quote the confidence score when summarizing.
Replace a brittle _cdp.py-style CSS-selector extractor
Before: hand-written CSS/XPath selectors break whenever the site redesigns.
After: feed research-fetch the URL; its VLM reconciliation adapts to DOM changes automatically. Pay the extra LLM cost once, keep extraction stable for months.
Related Skills
- chrome-devtools — interactive UI automation (click, fill, E2E). Complementary, not overlapping.
- deep-research — methodology for multi-source technical investigation. Use research-fetch as its per-URL fetcher.
