web-research

name

web-research

description

Correct protocol for crawling and researching external websites. Use this skill — always — before fetching any sub-page of a website you haven't visited before. Triggers on: "research this site", "check their website", "find out what X does", "scrape this competitor", "look at their features", "crawl this URL", "fetch pages from", "browse this site", "check what pages exist at", or any time you need information from multiple pages of the same domain. The rule is absolute: never guess or construct URL paths — always extract real hrefs from the homepage first.

Web Research — Correct Crawling Protocol

The Core Rule

Never construct or guess URL paths. Always extract real hrefs first.

This is not a preference — it is a hard rule with zero exceptions. URLs that look logical (/gestion-academica/, /facturacion-ventas/, /soluciones/) return 404 when the real paths are /gestion/, /facturacion/, /productos/. The site owns its URL structure, not you.

---

The Correct Protocol (always follow this order)

Step 1 — Fetch the root/homepage first

WebFetch(url: "https://domain.com/", prompt: "Extract EVERY href link on this page — all navigation links, footer links, button links, and any internal URL. I need the exact full URLs as they appear in the HTML so I can crawl them. List them all.")

This gives you the real URL structure of the site. Never skip this step.

Step 2 — Read the extracted URLs

From the homepage fetch, you now have a list of real paths. Use these — only these — for subsequent fetches.

Step 3 — Fetch the real pages in parallel

Now that you have confirmed URLs, fetch multiple pages simultaneously:

WebFetch(url: "https://domain.com/real-path-1/", prompt: "...")
WebFetch(url: "https://domain.com/real-path-2/", prompt: "...")
WebFetch(url: "https://domain.com/real-path-3/", prompt: "...")