crawl-budget-optimization

name	crawl-budget-optimization
description	Use when site has thousands of pages. Crawl budget = how often Google can crawl your site. Optimize for important pages.

Crawl Budget Optimization

Crawl budget = limited. Important sites: prioritize important pages. Block low-value pages (faceted search, filtered URLs).

Identifying crawl waste

Check Search Console > Crawl Stats:

1. URLs crawled per day
2. Average response time
3. Pages with 5xx errors (waste)
4. Pages with 404 (waste)
5. Redirects in crawl path (waste)
6. Faceted/filtered URLs (often waste)
7. Sorted/paginated infinite URLs (waste)

Crawl optimization tactics

1. Block low-value pages: faceted filters, search results, calendar variations
2. Fix errors: 5xx, 404s
3. Reduce redirects: 301 chains waste budget
4. Internal linking: pass priority signals to important pages
5. Sitemap priority: indicate page importance
6. Fast server response: faster TTFB = more URLs crawled per hour

When crawl budget matters

1. Sites with 1,000+ URLs
2. E-commerce with product catalog
3. News sites with deep archives
4. Sites with faceted search
5. International multi-language sites

Small sites (<500 URLs): crawl budget rarely matters.

Where this fits in the X3 empire

Important for X3 multi-product portfolio coordination.