seo-technical

name	seo-technical
description	Audit technical SEO across crawlability, indexability, security, URLs, mobile, Core Web Vitals, structured data, JavaScript rendering, and platform signals like robots.txt and AI-crawler access. USE WHEN a user wants a technical SEO review focused on crawl/index, performance, or rendering rather than content.
cluster	growth-seo
version	1.0.0
origin	antigravity-awesome-skills (MIT)
user-invokable	true
argument-hint	[url]
allowed-tools	["Read","Grep","Glob","Bash","WebFetch"]

Technical SEO Audit

When to Use

• Use when the user wants a technical SEO review focused on crawlability, indexability, performance, or rendering.
• Use when auditing robots.txt, canonicalization, JavaScript SEO, Core Web Vitals, or AI crawler access.
• Use when the task is infrastructure- and implementation-oriented rather than content-focused.

Categories

1. Crawlability

• robots.txt: exists, valid, not blocking important resources
• XML sitemap: exists, referenced in robots.txt, valid format
• Noindex tags: intentional vs accidental
• Crawl depth: important pages within 3 clicks of homepage
• JavaScript rendering: check if critical content requires JS execution
• Crawl budget: for large sites (>10k pages), efficiency matters

AI Crawler Management

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

Known AI crawlers:

Crawler	Company	robots.txt token	Purpose
GPTBot	OpenAI	GPTBot	Model training
ChatGPT-User	OpenAI	ChatGPT-User	Real-time browsing
ClaudeBot	Anthropic	ClaudeBot	Model training
PerplexityBot	Perplexity	PerplexityBot	Search index + training
Bytespider	ByteDance	Bytespider	Model training
Google-Extended	Google	Google-Extended	Gemini training (NOT search)
CCBot	Common Crawl	CCBot	Open dataset

Key distinctions:

• Blocking Google-Extended prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use Googlebot)
• Blocking GPTBot prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing ()