web-fetch

name	web-fetch
description	Web content fetching via curl and WebFetch when a specific URL is provided. Covers HTTP GET/POST, JSON APIs, HTML, auth, cookies. Triggers on: "fetch this URL", "download HTML", "call this API", "curl this endpoint". NOT for search, use tavily.
metadata	{"version":"1.1.1","category":"content","tags":["http","curl","api","web-content"],"difficulty":"beginner"}

Web Fetch

All web content retrieval uses curl (Bash) or the built-in WebFetch tool. No MCP server needed — Claude Code's native tools cover every Fetch MCP operation with more control.

Quick Reference

Fetch MCP Tool	Replacement	When to Use
fetch_html	curl -s URL	Raw HTML needed for parsing
fetch_json	curl -s URL | jq '.'	API responses, structured data
fetch_markdown	WebFetch	Readable page content (default output is markdown)
fetch_txt	curl -s URL or WebFetch	Plain text extraction

Default choice: Use WebFetch for general page content. Use curl when you need headers, authentication, POST bodies, or raw format control.

---

WebFetch (Built-in Tool)

The WebFetch tool fetches a URL and returns clean markdown content. It handles JavaScript-rendered pages, strips navigation and boilerplate, and returns readable text.

Best for: documentation pages, articles, blog posts, README files — any content where you want readable text rather than raw HTML.

Limitations: no custom headers, no POST bodies, no cookie management. Use curl for those.

---

curl Patterns

Fetch HTML

curl -sL "https://example.com/page"

Flag	Purpose
-s	Silent mode — suppress progress meter
-L	Follow redirects (3xx)
-o file.html	Save to file instead of stdout
-I	Headers only (HEAD request)
-i	Include response headers in output

Fetch and extract specific elements with xmllint or python3:

curl -sL "https://example.com" | python3 -c "
from html.parser import HTMLParser
import sys

class TitleParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.in_title = False
        self.title = ''
    def handle_starttag(self, tag, attrs):
        self.in_title = tag == 'title'
    def handle_data(self, data):
        if self.in_title:
            self.title += data
    def handle_endtag(self, tag):
        if tag == 'title':
            self.in_title = False

p = TitleParser()
p.feed(sys.stdin.read())
print(p.title)
"

Fetch JSON

curl -s "https://api.example.com/v1/data" \
  -H "Accept: application/json" | jq '.'

Filter and reshape JSON responses:

# Extract specific fields
curl -s "https://api.example.com/users" | jq '.[] | {name, email}'

# Filter by condition
curl -s "https://api.example.com/items" | jq '[.[] | select(.status == "active")]'

# Count results
curl -s "https://api.example.com/items" | jq 'length'

# Get nested value
curl -s "https://api.example.com/config" | jq '.database.host'

Fetch Plain Text

# Strip HTML tags for plain text
curl -sL "https://example.com/page" | python3 -c "
import html.parser, sys

class Stripper(html.parser.HTMLParser):
    def __init__(self):
        super().__init__()
        self.text = []
    def handle_data(self, d):
        self.text.append(d)
    def get_text(self):
        return ''.join(self.text)

s = Stripper()
s.feed(sys.stdin.read())
print(s.get_text())
"

Or use WebFetch which returns clean markdown — close enough to plain text for most purposes.

---

Authenticated Requests

Bearer Token

curl -s "https://api.example.com/data" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json"

API Key in Header

curl -s "https://api.example.com/data" \
  -H "X-API-Key: $API_KEY"

API Key in Query Parameter

curl -s "https://api.example.com/data?api_key=$API_KEY"

Basic Auth

curl -s -u "username:$PASSWORD" "https://api.example.com/data"

Store credentials in environment variables. Never hardcode tokens or passwords in commands.

---

POST, PUT, PATCH, DELETE

POST with JSON Body

curl -s -X POST "https://api.example.com/items" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_TOKEN" \
  -d '{
    "name": "item-name",
    "value": 42
  }' | jq '.'

POST with Form Data

curl -s -X POST "https://api.example.com/upload" \
  -F "file=@./document.pdf" \
  -F "description=Uploaded via curl"

PUT (Full Update)

curl -s -X PUT "https://api.example.com/items/123" \
  -H "Content-Type: application/json" \
  -d '{"name": "updated-name", "value": 99}' | jq '.'

PATCH (Partial Update)

curl -s -X PATCH "https://api.example.com/items/123" \
  -H "Content-Type: application/json" \
  -d '{"value": 100}' | jq '.'

DELETE

curl -s -X DELETE "https://api.example.com/items/123" \
  -H "Authorization: Bearer $API_TOKEN"

---

Advanced Patterns

Pagination

PAGE=1
while true; do
  RESPONSE=$(curl -s "https://api.example.com/items?page=$PAGE&per_page=50" \
    -H "Authorization: Bearer $API_TOKEN")
  COUNT=$(echo "$RESPONSE" | jq 'length')
  echo "$RESPONSE" | jq '.[]'
  [ "$COUNT" -lt 50 ] && break
  PAGE=$((PAGE + 1))
done

Timeout and Retry

curl -s --connect-timeout 10 --max-time 30 \
  --retry 3 --retry-delay 2 \
  "https://api.example.com/data"

Response Headers Inspection

curl -sI "https://example.com" | grep -i "content-type"

Save Response with Status Code

HTTP_CODE=$(curl -s -o /tmp/response.json -w "%{http_code}" "https://api.example.com/data")
echo "Status: $HTTP_CODE"
cat /tmp/response.json | jq '.'

Cookie Handling

# Save cookies
curl -s -c /tmp/cookies.txt "https://example.com/login" \
  -d "user=admin&pass=$PASSWORD"

# Reuse cookies
curl -s -b /tmp/cookies.txt "https://example.com/dashboard"

---

Error Handling

HTTP Status	Meaning	Resolution
301/302	Redirect	Add -L flag to follow
401	Unauthorized	Check token/credentials; verify env var is set
403	Forbidden	Insufficient permissions or IP restriction
404	Not Found	Verify URL path; resource may be deleted
429	Rate Limited	Respect Retry-After header; add delay between requests
500	Server Error	Retry once; if persistent, report upstream
SSL error	Certificate issue	Do not use -k (insecure) — fix the root cause
Timeout	Network/server slow	Increase --max-time; check connectivity

Verify a URL is reachable before complex operations:

curl -sI -o /dev/null -w "%{http_code}" "https://example.com"

---

Limitations

• WebFetch does not support custom headers, POST bodies, or cookies. Use curl for authenticated or stateful requests.
• curl does not render JavaScript. For JS-heavy SPAs, prefer WebFetch which handles rendered content.
• Large responses may exceed context limits. Pipe through jq, head, or python3 to extract only needed data before loading into context.
• Binary content (images, PDFs, archives) should be saved to disk with -o, not piped to stdout.

---

Calibration Rules

1. Default to WebFetch for reading web pages. It returns clean markdown, handles JS rendering, and requires no flags. Switch to curl only when you need headers, auth, POST, or raw format control.
2. Always pipe JSON through jq. Raw JSON in context wastes tokens. Filter to only the fields needed.
3. Never hardcode credentials. Use $ENV_VAR references. If the variable is not set, surface the error immediately.
4. Follow redirects by default. Always use -L with curl unless you specifically need to inspect the redirect chain.
5. Prefer -s (silent) on every curl call. Progress meters add noise to output.