fetch-403

name	fetch-403
argument-hint	[url that was refused]
description	Recover a web page after the built-in fetcher (WebFetch) is refused — 403 Forbidden, empty body, or a "Just a moment…" bot-check interstitial instead of content. Use when a documentation page, blog post, or reference URL won't load through the normal fetch tool and the content is still needed. Covers the curl fallback, machine-readable endpoints, and the rule against silently answering from memory when a source stays blocked.

Fetching a page that returned 403

A 403 from the built-in fetcher usually means that fetcher is unwelcome, not that the site blocks automation. debezium.io is the reference case: WebFetch gets 403, bare curl gets 200. WebFetch's 403 message may suggest the page needs authentication — try rung 1 before believing that; public docs pages 403 the fetcher's UA all the time.

Work down this list. Stop at the first rung that returns content.

1. curl

scripts/fetch403.sh URL [OUTFILE.md]   # default outfile: /tmp/page.md

Curls the URL, retries once with a browser User-Agent if that 403s (fixes plain UA filtering), converts to markdown, and prints the headings. Transient failures and cookie-then-redirect flows are handled inside curl; the markdown step falls back to raw HTML when neither html2text nor uvx is installed. Exits non-zero if the page stays blocked or comes back as a bot-check interstitial.

A docs page is often 3-4k lines of markdown. Read only the sections the headings point at with sed -n 'START,ENDp'; don't read the whole file into context.

2. The machine-readable endpoint

Many sites publish something built for programmatic reads. Prefer it — it is smaller, stabler, and never fights you:

• context7 MCP, when the blocked page is a library/framework/CLI doc — it has the content already extracted, so a 403 on the rendered page is irrelevant. resolve-library-id then query-docs, one concept per query, three calls max. It ranks by documentation coverage rather than recency, so read the Versions: list and pass /org/project/version when you need a specific release rather than whatever ranked first.
• llms.txt at the doc root (Confluent, Cloudflare, and a growing set)
• raw.githubusercontent.com for anything in a repo; gh api for private ones
• a JSON API, an RSS feed, or the .md source behind a rendered page

3. Search instead of fetch

Search snippets, a mirror, or the project's own repo often carry the answer. The goal is the content, not that specific URL.

4. Still blocked? Say so.

Report that the source was blocked and name what you tried. Never quietly answer from training memory instead — on a retrieval-first task that is the exact failure the retrieval was there to prevent, and it is invisible to the user. An unavailable source is a fact worth one sentence; a fabricated answer that looks sourced is not recoverable.

What not to reach for

Headless-browser scraping (Playwright/Puppeteer) does not clear this bar. Tested 2026-07 against a live Cloudflare challenge: vanilla Playwright returns an empty body, headless and headed, and costs ~646MB of browser download to do it. Worse, it exits 0 with empty output — a silent failure that reads as "the page was empty." Passing a real bot check needs stealth/fingerprint evasion, which is out of scope here. If a site fights that hard, it does not want automated reads: ask the user to paste the page.