| name | seo-technical |
| description | Audit technical SEO across crawlability, indexability, security, URLs, mobile, Core Web Vitals, structured data, JavaScript rendering, and platform signals like robots.txt and AI-crawler access. USE WHEN a user wants a technical SEO review focused on crawl/index, performance, or rendering rather than content. |
| cluster | growth-seo |
| version | 1.0.0 |
| origin | antigravity-awesome-skills (MIT) |
| user-invokable | true |
| argument-hint | [url] |
| allowed-tools | ["Read","Grep","Glob","Bash","WebFetch"] |
Technical SEO Audit
When to Use
- Use when the user wants a technical SEO review focused on crawlability, indexability, performance, or rendering.
- Use when auditing robots.txt, canonicalization, JavaScript SEO, Core Web Vitals, or AI crawler access.
- Use when the task is infrastructure- and implementation-oriented rather than content-focused.
Categories
1. Crawlability
- robots.txt: exists, valid, not blocking important resources
- XML sitemap: exists, referenced in robots.txt, valid format
- Noindex tags: intentional vs accidental
- Crawl depth: important pages within 3 clicks of homepage
- JavaScript rendering: check if critical content requires JS execution
- Crawl budget: for large sites (>10k pages), efficiency matters
AI Crawler Management
As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.
Known AI crawlers:
| Crawler | Company | robots.txt token | Purpose |
|---|
| GPTBot | OpenAI | GPTBot | Model training |
| ChatGPT-User | OpenAI | ChatGPT-User | Real-time browsing |
| ClaudeBot | Anthropic | ClaudeBot | Model training |
| PerplexityBot | Perplexity | PerplexityBot | Search index + training |
| Bytespider | ByteDance | Bytespider | Model training |
| Google-Extended | Google | Google-Extended | Gemini training (NOT search) |
| CCBot | Common Crawl | CCBot | Open dataset |
Key distinctions:
- Blocking
Google-Extended prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use Googlebot)
- Blocking
GPTBot prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing ()
