efficient-web-research

name	efficient-web-research
risk	safe
description	Protocol for token-efficient web research. Use when accessing URLs, GitHub repos, or running search queries. Prevents full-page fetching waste.

Efficient Web Research Skill

A protocol for accessing web content in the most token-efficient, accurate, and structured way — using the right tool at the right depth, and stopping as soon as the question is answerable.

---

When to Use

Use this skill when:

• Performing web searches, accessing external URLs, or reading GitHub repository details.
• Resolving technical details of trending libraries.
• Standardizing research protocols for token-efficiency.

Do NOT use this skill when:

• Executing local scripting or checking local directory layouts.

---

Core Principle

Fetch the minimum needed to answer. Skim before you dive. Stop when you can answer.

Every unnecessary fetch wastes tokens and adds noise. This skill enforces a layered approach where you escalate fetch depth only when shallower layers fail.

---

Step 1 — Classify the Input

Before fetching anything, identify what kind of input you received:

Input Type	Example	Go To
GitHub repo URL	github.com/user/repo	GitHub Protocol
Specific page URL	docs.python.org/3/library/os	URL Protocol
Topic / query (no URL)	"how does RAFT consensus work"	Search Protocol
Multiple URLs	List of links	Multi-URL Protocol
PDF / file link	.pdf, .txt, .md URL	File Protocol

---

GitHub Protocol

Use when input is a GitHub URL (repo, file, PR, issue, etc.)

Step 1 — Parse the URL

github.com/{owner}/{repo}                → Repo root
github.com/{owner}/{repo}/tree/{branch}  → Directory
github.com/{owner}/{repo}/blob/{branch}/{path} → Single file
github.com/{owner}/{repo}/issues/{n}     → Issue
github.com/{owner}/{repo}/pull/{n}       → Pull request

Step 2 — Use GitHub API (preferred over scraping)

Always prefer the GitHub API. It returns clean JSON — no HTML parsing needed.

# Repo metadata (name, description, language, stars, topics)
GET https://api.github.com/repos/{owner}/{repo}

# File tree (see what files exist — very cheap)
GET https://api.github.com/repos/{owner}/{repo}/git/trees/{ref}?recursive=1

# Single file content (base64 encoded)
GET https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={ref}

# README only (usually enough to understand the repo)
GET https://api.github.com/repos/{owner}/{repo}/readme

Step 3 — Layered Fetch for Repos

Layer 1 (always do first):
  → Fetch repo metadata + README only
  → Can you answer the user's question now? YES → STOP. NO → continue.

Layer 2 (only if needed):
  → Fetch file tree to understand structure
  → Identify the 1-3 most relevant files based on the question
  → Can you answer now? YES → STOP. NO → continue.

Layer 3 (last resort):
  → Fetch specific relevant files only (never fetch all files)
  → Prioritize: main entry point, config files, key modules

Token Rules for GitHub

• README alone answers ~70% of "what does this repo do" questions — always try it first
• Never fetch more than 3 files in a single research turn
• If a file exceeds ~300 lines, read only the top (imports + class/function signatures)
• Decode base64 content from API before passing to context

Step 4 — Reading & Analyzing Repos for Non-Technical Copywriting

When the goal is to write a non-technical/Socratic newsletter, do not read a repository like a developer looking to deploy it. Instead, read it like a translator looking for human pain points, core mechanisms, and relatable analogies.

1. Identify the Target Audience & Pain Point (Who & Why)

• Locate: Look at the top description, first paragraph, or sections named "Why", "Motivation", "Background", or "Problem".
• Look for: What makes a developer's life miserable without this tool? (e.g., slow build times, manual database backups, hard-to-debug networks).
• Filter: Ignore generic buzzwords like "fast", "scalable", or "developer-friendly". Extract the concrete situation where this tool saves the day.

2. Extract the Core Mechanism (How it works under the hood)

• Locate: Look at sections named "How it works", "Architecture", "Design", or "Core Features".
• Look for: The unique technology choice or design pattern that makes it work.
• Filter: Avoid listing CLI flags or config options. Focus on the core engine (e.g., "runs entirely in the browser via WebAssembly", "replicates data using WebRTC without a central server", "compiles raw SQL into REST endpoints").

3. Map to Lived Experience (Analogies & Metaphors)

• Brainstorming Socratic Analogies:
  • Look at the name of the repository or its logo for physical cues (e.g., a "bridge", a "gatekeeper", a "funnel").
  • Map the data flow to everyday concepts:
    • Cache/Buffer $\rightarrow$ Short-term memory, a refrigerator, a temporary desk tray.
    • Load Balancer $\rightarrow$ A traffic cop, a restaurant host, a distributor.
    • Database Replicant $\rightarrow$ A team of scribes copying books in real-time, backup backups.
    • Compiler/Parser $\rightarrow$ A multilingual translator, a recipe converter.
    • CLI Tool $\rightarrow$ A single swiss army knife vs. a heavy toolbox.

4. Handle Underspecified READMEs (Deep Dive Protocol)

• If the README has only a title and installation instructions, don't guess:
  • Check the File Tree:
    • Identify the programming language (e.g., Go, Rust, JavaScript).
    • Locate dependency files (package.json, Cargo.toml, go.mod) to see what external libraries it relies on.
    • Find the entry point file (e.g., main.go, index.js, app.py) to inspect the initialization logic.
  • Check the Examples Folder:
    • Look for /examples, /demo, /tests directories to see real-world usage patterns.
  • Check the Issues/PRs:
    • Look at the titles of recent PRs or resolved issues to understand what bugs or features the community is actively discussing.

---

URL Protocol

Use when the user gives a specific non-GitHub URL (docs, articles, blogs, etc.)

Step 1 — Assess the URL type

Site type	Likely works with	Notes
Static docs / MDN / ReadTheDocs	read_url_content	Fast, clean, cheap
News articles / blogs	read_url_content	Usually fine
SPAs / React/Next.js apps	browser_subagent	JS-rendered
Auth-gated pages	browser_subagent	Needs login
Raw GitHub files (raw.githubusercontent)	read_url_content	Direct text

Step 2 — Layered Fetch

Layer 1 — Skim
  → Fetch the URL with read_url_content
  → Read only headings (H1, H2, H3) and first paragraph
  → Does this page contain what the user needs? NO → try a different URL or search. YES → continue.

Layer 2 — Targeted Extract
  → If the page has anchor links (e.g. /docs/page#section), fetch with the anchor
  → Extract only the relevant section (200–500 tokens max)
  → Can you answer? YES → STOP.

Layer 3 — Full Fetch
  → Fetch full page, strip boilerplate (nav, footer, ads, cookie banners, sidebars)
  → Cap at 2000 tokens. Summarize before passing to answer.

Layer 4 — Browser Subagent (last resort only)
  → Use ONLY if read_url_content returns empty, garbled, or JS-placeholder content
  → Instruct subagent: "Navigate to [URL], wait for content to load, extract [specific section]"
  → Do NOT use browser_subagent for static pages — it's expensive

What to Strip from Fetched Pages

Always remove before using fetched content:

• Navigation menus and breadcrumbs
• Cookie banners and GDPR notices
• "Related articles" / "You might also like" blocks
• Footer content (copyright, links)
• Social share buttons
• Ads and sponsored content

Extract and keep:

• Main article / documentation body
• Code blocks
• Tables with data
• Numbered steps or procedures

---

Search Protocol

Use when the user gives a topic, question, or query — not a specific URL.

Step 1 — Sharpen the Query Before Searching

Do NOT search the raw user query. Transform it first:

Raw: "how to deploy fastapi on aws"
Sharpened: "fastapi AWS deployment tutorial 2024"

Raw: "python async vs threads"
Sharpened: "Python asyncio vs threading performance comparison"

Raw: "best way to structure react project"
Sharpened: "React project folder structure best practices"

Query sharpening rules:

• Add specificity: version numbers, technology names, "tutorial" / "guide" / "comparison"
• Add recency if relevant: current year
• Remove filler words: "how do I", "what is the", "can you explain"
• For code questions: add the language + framework name explicitly

Step 2 — Search and Select

1. Run search_web with the sharpened query
2. Get results (titles + snippets)
3. Scan titles + snippets ONLY — do not fetch yet
4. Pick the TOP 1-2 most relevant results (max 3 in complex cases)
5. Skip results from: forums (if docs exist), aggregator blogs, paywalled sites
6. Prefer: official docs, GitHub repos, well-known tech blogs, academic sources

Step 3 — Fetch Selected Results

Apply the URL Protocol (above) to each selected URL. Process results one at a time — only fetch the second URL if the first didn't answer the question.

Token Rules for Search

• Never read more than 3 URLs per search query
• If the snippet already contains the answer → do NOT fetch the full page, use the snippet
• For factual questions (dates, names, simple facts) → snippet is usually enough
• For procedural questions (how to do X) → fetch 1 relevant page, targeted section only

---

Multi-URL Protocol

Use when the user provides a list of URLs to compare or summarize.

1. Skim all URLs first (Layer 1 fetch for each)
2. Group by relevance to the user's question
3. Deep-fetch only the most relevant 1-3 URLs
4. Summarize each in 3-5 sentences before combining
5. Never dump raw content from multiple pages — always summarize per-source first

---

File Protocol

Use when URL points directly to a file (PDF, .txt, .md, .csv, etc.)

• .md / .txt / .csv → read_url_content works directly, read full content
• .pdf → Use browser_subagent or a PDF extraction tool; extract text only
• .json / .yaml → read_url_content, parse structure, summarize schema + key values
• Large files (>500 lines) → Read first 100 lines + last 20 lines + search for relevant sections

---

Anti-Patterns (Never Do These)

Anti-pattern	Why it's bad	Do this instead
Fetching full page for a simple fact	Wastes 1000s of tokens	Use snippet or targeted anchor
Using browser_subagent for static sites	Very expensive	Use read_url_content first
Searching with the raw user query	Vague results	Sharpen query first
Fetching 5+ search results	Token explosion	Max 3, stop when answered
Dumping raw HTML into context	Noisy, wasteful	Always strip to Markdown
Fetching "just in case"	Unnecessary tokens	Only fetch what's needed to answer
Re-fetching the same URL	Redundant	Cache result in context, reuse
Fetching entire GitHub repo	Extremely wasteful	README + targeted files only

---

Decision Flowchart (Quick Reference)

Input received
│
├─ GitHub URL?
│   ├─ Fetch README + metadata via API
│   ├─ Answered? → STOP
│   ├─ Need more? → Fetch file tree, pick 1-3 files
│   └─ Still need more? → Fetch specific files only
│
├─ Specific URL?
│   ├─ Try read_url_content → skim headings
│   ├─ Answered? → STOP
│   ├─ Need more? → Targeted section fetch
│   ├─ Still need more? → Full fetch, stripped
│   └─ JS-rendered / broken? → browser_subagent (last resort)
│
├─ Topic/query?
│   ├─ Sharpen query
│   ├─ search_web → scan snippets
│   ├─ Snippet enough? → Answer from snippet, STOP
│   ├─ Need more? → Fetch top 1 result (targeted)
│   └─ Still need more? → Fetch top 2nd result (targeted)
│
└─ List of URLs?
    ├─ Skim all (Layer 1 each)
    ├─ Deep fetch top 1-3 relevant ones
    └─ Summarize per-source, then combine

---

Output Format Rules

After fetching, structure your response as:

Source: [URL or "Web search for: query"]
Summary: [2-5 sentences of what was found]
Answer: [Direct answer to user's question]
Confidence: [High / Medium / Low — based on source quality]

For multiple sources:

Source 1: ...
Source 2: ...
Combined Answer: ...

Never output:

• Raw HTML fragments
• Full page dumps
• Unattributed information
• More than needed to answer the question

---

Token Budget Guide

Operation	Approximate token cost	When to use
GitHub README fetch	~300–800 tokens	Always first for repos
GitHub API metadata	~200 tokens	Always for repos
Skim (headings only)	~100–200 tokens	Always first for URLs
Targeted section fetch	~300–600 tokens	When skim isn't enough
Full page fetch (stripped)	~1000–2000 tokens	Only when targeted fails
browser_subagent	~2000–5000 tokens	Last resort only
Search snippet scan	~300–500 tokens	Always before fetching

Rule of thumb: If you're about to spend >2000 tokens on a fetch, ask yourself if there's a cheaper path first.

---

Limitations

• JavaScript Reliance: Standard fetching may not fully render Single Page Applications (SPAs). You must fallback to browser_subagent for these, which is slower and more expensive.
• Paywalls & Protections: This skill cannot bypass CAPTCHAs, bot protections (e.g., strict Cloudflare rules), or hard paywalls.
• GitHub API Limits: Frequent GitHub API requests without authentication may hit rate limits.