curl

name	cURL
description	cURL (curl): Command-line web browser alternative. Fetch webpage HTML, test site connectivity, bypass basic bot detection, and read public web content directly from the terminal.
metadata	{"version":"1.0.0","maxent":{"requires":{"bins":[{"name":"curl"}]}}}

cURL (curl)

Expert guidance for using cURL as a terminal-based internet browsing tool. Use cURL to quickly read web pages, fetch documentation, bypass basic bot detection by mimicking real browsers, and interact with web content when a visual browser is unnecessary.

Internet Browsing & Scraping

Fetching Web Pages

# Read the HTML content of a page directly to the terminal
curl https://example.com

# Follow redirects (crucial for modern web browsing, pages often redirect you)
curl -L https://example.com

# Silent mode with redirects (best for clean output when piping)
curl -sL https://example.com

Mimicking a Real Browser (User-Agent)

Many sites block default cURL requests. Spoof a standard browser User-Agent to retrieve content successfully and avoid 403 Forbidden errors.

# Browse as Chrome on Windows
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -L https://example.com

# Browse as Safari on macOS
curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15" -L https://example.com

Viewing Page Metadata

If you just want to see if a page exists or check what server/framework is being used without downloading the whole HTML body:

# Fetch only the HTTP headers (HEAD request)
curl -I -L https://example.com

# Print headers along with the body
curl -i -L https://example.com

Navigating Authenticated Web Content

# Send a cookie when attempting to read behind a login wall
curl --cookie "sessionid=xyz123" -L https://example.com/dashboard

# Download cookies from a login page to use in subsequent requests
curl --cookie-jar cookies.txt -L https://example.com/login
curl --cookie cookies.txt -L https://example.com/protected-page

Bypassing Security Flags

# Ignore SSL certificate warnings (useful for older sites or local dev environments)
curl -k -L https://untrusted-ssl.example.com

Extracting Content (Piping)

Since cURL returns raw HTML, you can pipe it to other built-in bash tools to filter out the noise and read the relevant information.

# Fetch a page and search for specific content
curl -sL https://example.com | grep -i "specific text"

# Fetch a page and strip HTML tags to read plain text (basic approach)
curl -sL https://example.com | sed -e 's/<[^>]*>//g'

Downloading Web Assets

While browsing, you may want to download standard assets like images, PDFs, or tarballs directly.

# Download a file and keep its original name
curl -O https://example.com/assets/report.pdf

# Save an entire webpage to a file to analyze offline
curl -o index.html -L https://example.com

REST API & Service Interaction

Sometimes sites provide data endpoints which are easier to read than HTML.

# Fetch JSON data from an endpoint
curl -sL -H "Accept: application/json" https://api.example.com/data