web-research-fallback

# Web Research Fallback Patterns ## Cuándo usar - Cuando todos los motores de búsqueda (Google, DuckDuckGo, Bing) bloquean con CAPTCHA desde la IP del servidor - Cuando se necesita investigar información técnica y no hay acceso a buscadores convencionales - Cuando se requiere consultar Wikipedia/Wikidata como fuente fiable desde un entorno servidor - Cuando se necesita extraer contenido de sitios web directos sin pasar por un buscador ## Cuándo NO usar - Cuando los buscadores convencionales funcionan correctamente → usarlos directamente (mejores resultados) - Cuando se necesita investigación interactiva con el usuario → preguntar al usuario por URLs específicas - Cuando la información buscada no está disponible en fuentes públicas → pedir detalles al usuario - Cuando se necesita análisis de código o auditoría → usar `web-audit` o `dogfood` en su lugar ## Problem When running from a server (VM, container, CI), all major search engines block with CAPTCHA: - **Google**: IP-based CAPTCHA challenge - **DuckDuckGo**: "Select all squares containing a duck" image CAPTCHA - **Bing**: Returns HTML but with different structure (Finnish layout, no b_algo class) - **SearXNG public instances**: All currently down or returning empty ## Fallback Chain (try in order) ### 1. Wikipedia/Wikidata APIs (most reliable) ```bash # Wikipedia search curl -s "https://es.wikipedia.org/w/api.php?action=query&list=search&srsearch=QUERY&format=json&srlimit=5" # Wikipedia full article extract curl -s "https://es.wikipedia.org/w/api.php?action=query&prop=extracts&explaintext&titles=ARTICLE_NAME&format=json" # Wikidata search curl -s "https://www.wikidata.org/w/api.php?action=wbsearchterms&search=QUERY&language=es&format=json&limit=5" ``` **Tips:** - Use Spanish Wikipedia (`es.wikipedia.org`) for Spanish queries — better results for Spanish entities - `explaintext` returns plain text without HTML markup - `exintro` returns only the first section (intro) - Wikidata `wbsearchterms` returns structured data with labels and descriptions - Cross-reference multiple Wikipedia language versions (es, en, eu, fr) ### 2. Direct site navigation (browser tool) When you know the target website: 1. Navigate directly to the URL 2. Use `browser_snapshot(full=true)` for full page content 3. Use `browser_console(expression="document.body.innerText")` to get clean text 4. Use `browser_console(expression="JSON.stringify(Array.from(document.querySelectorAll('a')).map(a => ({text: a.innerText.trim(), href: a.href})))")` to extract all links 5. Use `browser_vision(question=...)` for visual understanding of complex layouts 6. For SPAs (Single Page Applications), the DOM may not match the URL — always check `document.body.innerText` for actual content **SPA-specific pitfalls:** - SPA navigation doesn't change the URL — the content is loaded dynamically - Footer links may point to different routes than expected (e.g., "Development center" → `/crear`) - Use `browser_console` to extract all links and find the real URLs - `browser_snapshot` may show only the navigation shell if the SPA hasn't rendered yet **⚠️ Browser tool environment failures:** - If `browser_navigate` fails with undici/Node.js module errors (e.g., `Cannot load externalized builtin: "internal/deps/undici/undici-fetch.js"`), the browser tool is broken in this environment. Fall through to Step 3 (curl) or Step 4 (Wikipedia API). - This is an environment-specific failure, not a permanent browser limitation. The browser tool may work in other environments. ### 3. Curl with proper headers ```bash curl -s -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" \ -H "Accept-Language: es-ES,es;q=0.9" \ URL ``` **Best pattern for research tasks:** Download to file first, then parse with `execute_code`: ```bash curl -s -A "Mozilla/5.0 ..." URL -o /tmp/page.html ``` Then use `execute_code` to parse the HTML with Python (regex, BeautifulSoup, lxml). This avoids the `curl | python3` pipe security scan. **RSS feeds are the best source for trend research:** Most tech blogs expose RSS/Atom feeds with structured content. Parse them with regex or XML parsing. Examples: - CSS-Tricks: `https://css-tricks.com/feed/` - MDN Blog: `https://developer.mozilla.org/en-US/blog/rss.xml` - CSS-Tricks articles: extract `<title>`, `<pubDate>`, `<link>`, and CDATA descriptions **Bing-specific:** Bing returns HTML but with different class names than expected. The `b_algo` class may not exist. Instead, look for `<h2><a>` patterns or parse the raw HTML more flexibly. ### 3b. RSS-first research pattern (proven for CSS/web dev) When researching web development trends, RSS feeds are more reliable than HTML page scraping: 1. Fetch the RSS feed URL with curl 2. Parse XML for `<item>` entries 3. Extract `<title>`, `<pubDate>`, `<link>`, and CDATA `<description>` from each item 4. For articles of interest, fetch the full article URL with curl 5. Extract `<article>` content with regex, strip HTML tags 6. Use `execute_code` for parsing (not `curl | python3` pipe) This pattern worked for CSS-Tricks (15 articles parsed in one fetch) and MDN Blog. ### 4. Wikipedia in other languages When Spanish Wikipedia has no results, try: - English Wikipedia (`en.wikipedia.org`) - Basque Wikipedia (`eu.wikipedia.org`) — useful for Basque entities - French Wikipedia (`fr.wikipedia.org`) ### 5. Direct knowledge (when research fails) If all technical research fails: - Use what you know about the domain - Ask the user for specific details (URLs, context, keywords) - Create the deliverable with clearly marked assumptions - The user can fill in gaps from their knowledge ## Common Pitfalls - **CAPTCHA from server IPs**: Always expected. Don't retry the same search engine. Move to the next fallback. - **Spanish government sites block server IPs**: INE (ine.es), datos.gob.es, Idealista, Fotocasa, Portal de Vivienda ALL block curl from server IPs regardless of headers. This is a persistent pattern, not a transient failure. See skill `spanish-open-data-collection` for the full blocked sources list and estimation workaround. - **SPA navigation**: The URL bar doesn't reflect actual content. Always check `document.body.innerText`. - **Empty curl responses**: Could be CAPTCHA, could be the site requiring JS rendering. Try the browser tool. - **Bing returns Finnish layout**: This is a geo-blocking artifact. The HTML structure may differ. Don't assume `b_algo` class exists. - **DuckDuckGo lite returns CAPTCHA**: The HTML endpoint (`html.duckduckgo.com`) also blocks server IPs. ## When to Ask the User If you've tried the full fallback chain (Wikipedia API → Wikidata → direct site navigation → curl with headers) and still can't find the information, ask the user for: - Specific URLs to check - Keywords or context they can provide - Whether they can do a quick search and share results