| name | efficient-web-research |
| risk | safe |
| description | Protocol for token-efficient web research. Use when accessing URLs, GitHub repos, or running search queries. Prevents full-page fetching waste.
|
Efficient Web Research Skill
A protocol for accessing web content in the most token-efficient, accurate, and structured way —
using the right tool at the right depth, and stopping as soon as the question is answerable.
When to Use
Use this skill when:
- Performing web searches, accessing external URLs, or reading GitHub repository details.
- Resolving technical details of trending libraries.
- Standardizing research protocols for token-efficiency.
Do NOT use this skill when:
- Executing local scripting or checking local directory layouts.
Core Principle
Fetch the minimum needed to answer. Skim before you dive. Stop when you can answer.
Every unnecessary fetch wastes tokens and adds noise. This skill enforces a layered approach
where you escalate fetch depth only when shallower layers fail.
Step 1 — Classify the Input
Before fetching anything, identify what kind of input you received:
GitHub Protocol
Use when input is a GitHub URL (repo, file, PR, issue, etc.)
Step 1 — Parse the URL
github.com/{owner}/{repo} → Repo root
github.com/{owner}/{repo}/tree/{branch} → Directory
github.com/{owner}/{repo}/blob/{branch}/{path} → Single file
github.com/{owner}/{repo}/issues/{n} → Issue
github.com/{owner}/{repo}/pull/{n} → Pull request
Step 2 — Use GitHub API (preferred over scraping)
Always prefer the GitHub API. It returns clean JSON — no HTML parsing needed.
# Repo metadata (name, description, language, stars, topics)
GET https://api.github.com/repos/{owner}/{repo}
# File tree (see what files exist — very cheap)
GET https://api.github.com/repos/{owner}/{repo}/git/trees/{ref}?recursive=1
# Single file content (base64 encoded)
GET https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={ref}
# README only (usually enough to understand the repo)
GET https://api.github.com/repos/{owner}/{repo}/readme
Step 3 — Layered Fetch for Repos
Layer 1 (always do first):
→ Fetch repo metadata + README only
→ Can you answer the user's question now? YES → STOP. NO → continue.
Layer 2 (only if needed):
→ Fetch file tree to understand structure
→ Identify the 1-3 most relevant files based on the question
→ Can you answer now? YES → STOP. NO → continue.
Layer 3 (last resort):
→ Fetch specific relevant files only (never fetch all files)
→ Prioritize: main entry point, config files, key modules
Token Rules for GitHub
- README alone answers ~70% of "what does this repo do" questions — always try it first
- Never fetch more than 3 files in a single research turn
- If a file exceeds ~300 lines, read only the top (imports + class/function signatures)
- Decode base64 content from API before passing to context
Step 4 — Reading & Analyzing Repos for Non-Technical Copywriting
When the goal is to write a non-technical/Socratic newsletter, do not read a repository like a developer looking to deploy it. Instead, read it like a translator looking for human pain points, core mechanisms, and relatable analogies.
1. Identify the Target Audience & Pain Point (Who & Why)
- Locate: Look at the top description, first paragraph, or sections named "Why", "Motivation", "Background", or "Problem".
- Look for: What makes a developer's life miserable without this tool? (e.g., slow build times, manual database backups, hard-to-debug networks).
- Filter: Ignore generic buzzwords like "fast", "scalable", or "developer-friendly". Extract the concrete situation where this tool saves the day.
2. Extract the Core Mechanism (How it works under the hood)
- Locate: Look at sections named "How it works", "Architecture", "Design", or "Core Features".
- Look for: The unique technology choice or design pattern that makes it work.
- Filter: Avoid listing CLI flags or config options. Focus on the core engine (e.g., "runs entirely in the browser via WebAssembly", "replicates data using WebRTC without a central server", "compiles raw SQL into REST endpoints").
3. Map to Lived Experience (Analogies & Metaphors)
- Brainstorming Socratic Analogies:
- Look at the name of the repository or its logo for physical cues (e.g., a "bridge", a "gatekeeper", a "funnel").
- Map the data flow to everyday concepts:
- Cache/Buffer $\rightarrow$ Short-term memory, a refrigerator, a temporary desk tray.
- Load Balancer $\rightarrow$ A traffic cop, a restaurant host, a distributor.
- Database Replicant $\rightarrow$ A team of scribes copying books in real-time, backup backups.
- Compiler/Parser $\rightarrow$ A multilingual translator, a recipe converter.
- CLI Tool $\rightarrow$ A single swiss army knife vs. a heavy toolbox.
4. Handle Underspecified READMEs (Deep Dive Protocol)
- If the README has only a title and installation instructions, don't guess:
- Check the File Tree:
- Identify the programming language (e.g., Go, Rust, JavaScript).
- Locate dependency files (
package.json, Cargo.toml, go.mod) to see what external libraries it relies on.
- Find the entry point file (e.g.,
main.go, index.js, app.py) to inspect the initialization logic.
- Check the Examples Folder:
- Look for
/examples, /demo, /tests directories to see real-world usage patterns.
- Check the Issues/PRs:
- Look at the titles of recent PRs or resolved issues to understand what bugs or features the community is actively discussing.
URL Protocol
Use when the user gives a specific non-GitHub URL (docs, articles, blogs, etc.)
Step 1 — Assess the URL type
| Site type | Likely works with | Notes |
|---|
| Static docs / MDN / ReadTheDocs | read_url_content | Fast, clean, cheap |
| News articles / blogs | read_url_content | Usually fine |
| SPAs / React/Next.js apps | browser_subagent | JS-rendered |
| Auth-gated pages | browser_subagent | Needs login |
| Raw GitHub files (raw.githubusercontent) | read_url_content | Direct text |
Step 2 — Layered Fetch
Layer 1 — Skim
→ Fetch the URL with read_url_content
→ Read only headings (H1, H2, H3) and first paragraph
→ Does this page contain what the user needs? NO → try a different URL or search. YES → continue.
Layer 2 — Targeted Extract
→ If the page has anchor links (e.g. /docs/page#section), fetch with the anchor
→ Extract only the relevant section (200–500 tokens max)
→ Can you answer? YES → STOP.
Layer 3 — Full Fetch
→ Fetch full page, strip boilerplate (nav, footer, ads, cookie banners, sidebars)
→ Cap at 2000 tokens. Summarize before passing to answer.
Layer 4 — Browser Subagent (last resort only)
→ Use ONLY if read_url_content returns empty, garbled, or JS-placeholder content
→ Instruct subagent: "Navigate to [URL], wait for content to load, extract [specific section]"
→ Do NOT use browser_subagent for static pages — it's expensive
What to Strip from Fetched Pages
Always remove before using fetched content:
- Navigation menus and breadcrumbs
- Cookie banners and GDPR notices
- "Related articles" / "You might also like" blocks
- Footer content (copyright, links)
- Social share buttons
- Ads and sponsored content
Extract and keep:
- Main article / documentation body
- Code blocks
- Tables with data
- Numbered steps or procedures
Search Protocol
Use when the user gives a topic, question, or query — not a specific URL.
Step 1 — Sharpen the Query Before Searching
Do NOT search the raw user query. Transform it first:
Raw: "how to deploy fastapi on aws"
Sharpened: "fastapi AWS deployment tutorial 2024"
Raw: "python async vs threads"
Sharpened: "Python asyncio vs threading performance comparison"
Raw: "best way to structure react project"
Sharpened: "React project folder structure best practices"
Query sharpening rules:
- Add specificity: version numbers, technology names, "tutorial" / "guide" / "comparison"
- Add recency if relevant: current year
- Remove filler words: "how do I", "what is the", "can you explain"
- For code questions: add the language + framework name explicitly
Step 2 — Search and Select
1. Run search_web with the sharpened query
2. Get results (titles + snippets)
3. Scan titles + snippets ONLY — do not fetch yet
4. Pick the TOP 1-2 most relevant results (max 3 in complex cases)
5. Skip results from: forums (if docs exist), aggregator blogs, paywalled sites
6. Prefer: official docs, GitHub repos, well-known tech blogs, academic sources
Step 3 — Fetch Selected Results
Apply the URL Protocol (above) to each selected URL.
Process results one at a time — only fetch the second URL if the first didn't answer the question.
Token Rules for Search
- Never read more than 3 URLs per search query
- If the snippet already contains the answer → do NOT fetch the full page, use the snippet
- For factual questions (dates, names, simple facts) → snippet is usually enough
- For procedural questions (how to do X) → fetch 1 relevant page, targeted section only
Multi-URL Protocol
Use when the user provides a list of URLs to compare or summarize.
1. Skim all URLs first (Layer 1 fetch for each)
2. Group by relevance to the user's question
3. Deep-fetch only the most relevant 1-3 URLs
4. Summarize each in 3-5 sentences before combining
5. Never dump raw content from multiple pages — always summarize per-source first
File Protocol
Use when URL points directly to a file (PDF, .txt, .md, .csv, etc.)
.md / .txt / .csv → read_url_content works directly, read full content.pdf → Use browser_subagent or a PDF extraction tool; extract text only.json / .yaml → read_url_content, parse structure, summarize schema + key values- Large files (>500 lines) → Read first 100 lines + last 20 lines + search for relevant sections
Anti-Patterns (Never Do These)
| Anti-pattern | Why it's bad | Do this instead |
|---|
| Fetching full page for a simple fact | Wastes 1000s of tokens | Use snippet or targeted anchor |
| Using browser_subagent for static sites | Very expensive | Use read_url_content first |
| Searching with the raw user query | Vague results | Sharpen query first |
| Fetching 5+ search results | Token explosion | Max 3, stop when answered |
| Dumping raw HTML into context | Noisy, wasteful | Always strip to Markdown |
| Fetching "just in case" | Unnecessary tokens | Only fetch what's needed to answer |
| Re-fetching the same URL | Redundant | Cache result in context, reuse |
| Fetching entire GitHub repo | Extremely wasteful | README + targeted files only |
Decision Flowchart (Quick Reference)
Input received
│
├─ GitHub URL?
│ ├─ Fetch README + metadata via API
│ ├─ Answered? → STOP
│ ├─ Need more? → Fetch file tree, pick 1-3 files
│ └─ Still need more? → Fetch specific files only
│
├─ Specific URL?
│ ├─ Try read_url_content → skim headings
│ ├─ Answered? → STOP
│ ├─ Need more? → Targeted section fetch
│ ├─ Still need more? → Full fetch, stripped
│ └─ JS-rendered / broken? → browser_subagent (last resort)
│
├─ Topic/query?
│ ├─ Sharpen query
│ ├─ search_web → scan snippets
│ ├─ Snippet enough? → Answer from snippet, STOP
│ ├─ Need more? → Fetch top 1 result (targeted)
│ └─ Still need more? → Fetch top 2nd result (targeted)
│
└─ List of URLs?
├─ Skim all (Layer 1 each)
├─ Deep fetch top 1-3 relevant ones
└─ Summarize per-source, then combine
Output Format Rules
After fetching, structure your response as:
Source: [URL or "Web search for: query"]
Summary: [2-5 sentences of what was found]
Answer: [Direct answer to user's question]
Confidence: [High / Medium / Low — based on source quality]
For multiple sources:
Source 1: ...
Source 2: ...
Combined Answer: ...
Never output:
- Raw HTML fragments
- Full page dumps
- Unattributed information
- More than needed to answer the question
Token Budget Guide
| Operation | Approximate token cost | When to use |
|---|
| GitHub README fetch | ~300–800 tokens | Always first for repos |
| GitHub API metadata | ~200 tokens | Always for repos |
| Skim (headings only) | ~100–200 tokens | Always first for URLs |
| Targeted section fetch | ~300–600 tokens | When skim isn't enough |
| Full page fetch (stripped) | ~1000–2000 tokens | Only when targeted fails |
| browser_subagent | ~2000–5000 tokens | Last resort only |
| Search snippet scan | ~300–500 tokens | Always before fetching |
Rule of thumb: If you're about to spend >2000 tokens on a fetch, ask yourself if there's a cheaper path first.
Limitations
- JavaScript Reliance: Standard fetching may not fully render Single Page Applications (SPAs). You must fallback to
browser_subagent for these, which is slower and more expensive.
- Paywalls & Protections: This skill cannot bypass CAPTCHAs, bot protections (e.g., strict Cloudflare rules), or hard paywalls.
- GitHub API Limits: Frequent GitHub API requests without authentication may hit rate limits.
