Use this skill when working on technical SEO infrastructure - crawlability, indexing, XML sitemaps, canonical URLs, robots.txt, redirect chains, rendering strategies (SSR/SSG/ISR/CSR), crawl budget optimization, and search engine rendering. Triggers on fixing indexing issues, configuring crawl directives, choosing rendering strategies for SEO, debugging Google Search Console errors, or auditing site architecture for search engines.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Use this skill when working on technical SEO infrastructure - crawlability, indexing, XML sitemaps, canonical URLs, robots.txt, redirect chains, rendering strategies (SSR/SSG/ISR/CSR), crawl budget optimization, and search engine rendering. Triggers on fixing indexing issues, configuring crawl directives, choosing rendering strategies for SEO, debugging Google Search Console errors, or auditing site architecture for search engines.
When this skill is activated, always start your first response with the 🧢 emoji.
Technical SEO
The infrastructure layer of SEO. Technical SEO ensures search engines can discover,
crawl, render, and index your pages. It is the foundation - if crawling fails, content
quality and link building are irrelevant. This skill covers the crawl-index-rank
pipeline and the engineering decisions that make or break search visibility.
When to use this skill
Trigger this skill when the user:
Reports pages not showing in Google Search or Index Coverage errors in Search Console
Needs to configure or debug robots.txt directives
Wants to generate or fix an XML sitemap
Is setting up canonical URLs or resolving duplicate content issues
Has redirect chains or wants to audit redirects
Is choosing a rendering strategy (SSR, SSG, ISR, CSR) with SEO as a constraint
Is debugging why Googlebot cannot see content that users can
Wants to optimize crawl budget on a large site (10k+ pages)
Do NOT trigger this skill for:
Content strategy, editorial calendars, or keyword research
Link building, backlink analysis, or off-page SEO
Key principles
Crawlable before rankable - A page that Googlebot cannot reach cannot rank.
Discovery is step one in the pipeline. Fix crawl and index issues before any
other SEO work. Crawlability is a precondition, not a ranking factor.
One canonical URL per piece of content - Every distinct piece of content
must have exactly one URL that all signals consolidate on. HTTP vs HTTPS,
www vs non-www, trailing slash vs none, query parameters - each variant dilutes
ranking signals unless canonicalized to a single source of truth.
Rendering strategy is an SEO architecture decision - Whether your page is
rendered at build time (SSG), at request time on the server (SSR), or in the
browser (CSR) determines whether Googlebot sees your content on the first crawl
or must wait for a second-wave JavaScript render. Make this decision deliberately.
robots.txt blocks crawling, not indexing - A page blocked in robots.txt
can still be indexed if other pages link to it. Googlebot sees the URL via links
but cannot read the content, so it may index a thin or empty page. Use noindex
in the HTTP response header or meta tag to prevent indexing, not robots.txt.
Redirect chains waste crawl budget and dilute link equity - Each hop in a
redirect chain costs crawl budget and reduces the link equity passed through.
Keep all redirects as single-hop 301s from old URL directly to final destination.
Core concepts
The crawl-index-rank pipeline
Three sequential phases - failure in any phase stops everything downstream:
Crawl budget is the number of URLs Googlebot will crawl on your site within a given
timeframe. It is a product of crawl rate (how fast Googlebot can crawl without
overloading the server) and crawl demand (how much Google wants to crawl based
on page value and freshness).
Who needs to care about crawl budget:
Sites with 10k+ pages
Sites with large faceted navigation generating URL permutations
Sites with many low-value or duplicate URLs (pagination, filters, sessions in URLs)
Sites with frequent content updates that need fast re-indexing
Small sites (<1k pages) with clean architecture rarely face crawl budget problems.
Rendering for crawlers
Googlebot can execute JavaScript but does so in a second wave, sometimes days after
the initial crawl. Content invisible without JavaScript is at risk:
Rendering
Googlebot sees on first crawl
SEO risk
SSG (static)
Full HTML
None
SSR (server-side)
Full HTML
None
ISR (incremental static)
Full HTML (on cache hit)
Minor - stale cache shows old content
CSR (client-side only)
Empty shell
High - content may not be indexed
URL parameter handling
URL parameters are a major source of duplicate content. Common problematic patterns:
Handle with: canonical tags pointing to the clean URL, robots.txt Disallow for
pure tracking parameters, or Google Search Console parameter handling.
Mobile-first indexing
Google indexes and ranks primarily based on the mobile version of your content.
Ensure the mobile version has: the same content as desktop, the same structured
data, and equivalent meta tags. Blocked mobile CSS/JS is a common cause of
mobile-first indexing failures.
Common tasks
Configure robots.txt
# Allow all crawlers to access all content (default, no file needed)
User-agent: *
Allow: /
# Block specific directories from all crawlers
User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Disallow: /checkout/
Disallow: /?*sessionid= # block session ID URLs
# Allow Googlebot to crawl CSS and JS (critical - never block these)
User-agent: Googlebot
Allow: /*.js$
Allow: /*.css$
# Point to sitemap
Sitemap: https://example.com/sitemap.xml
Never disallow CSS or JS. Googlebot needs them to render your pages. Blocking
them degrades rendering quality and can hurt rankings.
Sitemap rules: max 50,000 URLs per file, max 50MB uncompressed. Only include
canonical, indexable URLs. Only include lastmod if it reflects genuine content
changes - Googlebot learns to ignore dishonest lastmod values.
<!-- All of these should resolve to one canonical form --><!-- https://example.com/products/widget/ --><!-- https://example.com/products/widget --><!-- http://example.com/products/widget --><!-- https://www.example.com/products/widget --><!-- All pages declare the same canonical --><linkrel="canonical"href="https://example.com/products/widget" />
For paginated pages, each page is canonically itself (do not canonical page 2 to
page 1 unless they have identical content):
Decision table for ranking pages (pages you want to appear in search):
Content type
Recommended strategy
Rationale
Marketing pages, landing pages
SSG
Crawled immediately, fast TTFB
Blog posts, documentation
SSG
Rarely changes, build on publish
Product pages (10k-100k)
ISR
Manageable builds, auto-updates
User profiles, social content
SSR
Personalized but crawlable
Search results, filters
SSR + canonical
Crawlable canonical version
Dashboards, account pages
CSR is fine
Behind auth, not indexed anyway
For Next.js:
// SSG - crawled immediately, best for ranking pagesexportasyncfunctiongenerateStaticParams() { ... }
// ISR - rebuilds on demand, good for large catalogsexportconst revalidate = 3600; // revalidate every hour// SSR - server renders on every requestexportconst dynamic = 'force-dynamic';
Fix redirect chains
Redirect chains occur when A -> B -> C instead of A -> C directly. Detect and fix:
Option A is preferred when the canonical page has good content. Option B is
useful when you want to conserve crawl budget. Option C is the fallback when
you need to serve the page to users but not have it indexed.
Set up meta robots directives
In the HTML <head>:
<!-- Default: crawl and index (no tag needed) --><metaname="robots"content="index, follow" /><!-- Do not index, but follow links on this page --><metaname="robots"content="noindex, follow" /><!-- Do not index, do not follow links --><metaname="robots"content="noindex, nofollow" /><!-- Prevent Google from showing a cached version --><metaname="robots"content="index, follow, noarchive" />
Via HTTP response header (works for non-HTML resources like PDFs):
When a page is not indexed, work through this checklist in order:
URL Inspection tool in Search Console - checks crawl status, last crawl,
indexing decision, and renders a screenshot of what Googlebot sees
robots.txt tester - confirm the URL is not blocked
Live URL test - request indexing and see if Googlebot can render the page
Check for noindex - view source and search for noindex, check HTTP headers
Check canonical - is the canonical pointing to a different URL?
Check content - is there enough unique, substantive content?
Check internal links - is the page linked from anywhere Googlebot can reach?
Anti-patterns / common mistakes
Mistake
Why it is wrong
What to do instead
Blocking CSS/JS in robots.txt
Googlebot cannot render pages, sees empty shells
Allow: /*.js$ and Allow: /*.css$ explicitly
Dishonest lastmod in sitemap
Googlebot learns to ignore it; all URLs get low-priority crawls
Only update lastmod on genuine content changes
CSR-only rendering for rankable pages
Content in JS is not seen on first crawl; delayed or failed indexing
Use SSG or SSR for any page you want in search results
Client-side redirects for SEO
Meta refresh and JS redirects do not reliably pass link equity
Redirect at server/CDN level with 301
Using robots.txt to prevent indexing
Blocked pages can still be indexed as empty/thin if linked to
Use noindex directive in response headers or meta tag
Self-referential canonical loops
Page A canonicals to B, B canonicals to A; Google ignores both
Each URL canonicals to a single definitive URL
Duplicate canonicals pointing to 404s
Signals to Google the canonical URL is invalid
Ensure canonical targets return 200 with real content
Trailing slash inconsistency
Two URLs for every page, dilutes crawl budget and link signals
Enforce one form at the server, canonical the other
Noindex on paginated pages in series
First page gets indexed without context of full series
Only noindex pagination if pages are truly thin/duplicate
Sitemap URLs not matching canonicals
Confuses Googlebot about which URL is authoritative
Sitemap URLs must exactly match their canonical <link> tag
Gotchas
Canonical tags are a hint, not a directive - Google can and does override canonical tags if it disagrees with your signal. If your site serves near-identical content at two URLs and one has more internal links, Google may index the more-linked URL regardless of your canonical. Canonical must be reinforced with consistent internal linking and redirects.
Blocking CSS/JS in robots.txt causes Googlebot to see a broken page - Even a single blocked CSS file can prevent Googlebot from rendering a page correctly, leading to indexing of an empty or unstyled shell. Always test with the URL Inspection tool's "Test Live URL" option to confirm Googlebot's rendered view.
noindex in robots.txt does not work - The noindex directive in robots.txt is not a valid robots.txt directive; it is ignored. The only valid noindex placement is in the HTTP response header (X-Robots-Tag) or the HTML <meta name="robots"> tag on the page itself.
Hreflang is validated in both directions - If page A in English points to page B in French, page B must also point back to page A. Missing the return link causes Google to ignore the entire hreflang cluster. Validate every hreflang implementation bidirectionally.
Sitemap lastmod dates that never change train Googlebot to deprioritize your site - Many CMSs emit the current date as lastmod on every page regardless of actual changes. Googlebot learns these dates are dishonest and reduces crawl frequency. Only emit lastmod when content genuinely changed.
References
For detailed implementation guidance, load the relevant reference file:
Only load a reference file if the current task requires it - they are long and
will consume context.
Companion check
On first activation of this skill in a conversation: check which companion skills are installed by running ls ~/.claude/skills/ ~/.agent/skills/ ~/.agents/skills/ .claude/skills/ .agent/skills/ .agents/skills/ 2>/dev/null. Compare the results against the recommended_skills field in this file's frontmatter. For any that are missing, mention them once and offer to install:
