seo-technical

name	seo-technical
description	Audit technical SEO across crawlability, indexability, security, URLs, mobile, Core Web Vitals, structured data, JavaScript rendering, and related platform signals like robots.txt and AI crawler access.
type	skill
created	2026-02-27T00:00:00.000Z
domain	business-marketing
category	marketing
risk	unknown
source	https://github.com/AgriciDaniel/claude-seo
tags	["skill","business-marketing","marketing","seo","technical"]

Technical SEO Audit

When to Use

• Use when the user wants a technical SEO review focused on crawlability, indexability, performance, or rendering.
• Use when auditing robots.txt, canonicalization, JavaScript SEO, Core Web Vitals, or AI crawler access.
• Use when the task is infrastructure- and implementation-oriented rather than content-focused.

Categories

1. Crawlability

• robots.txt: exists, valid, not blocking important resources
• XML sitemap: exists, referenced in robots.txt, valid format
• Noindex tags: intentional vs accidental
• Crawl depth: important pages within 3 clicks of homepage
• JavaScript rendering: check if critical content requires JS execution
• Crawl budget: for large sites (>10k pages), efficiency matters

AI Crawler Management

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

Known AI crawlers:

Crawler	Company	robots.txt token	Purpose
GPTBot	OpenAI	GPTBot	Model training
ChatGPT-User	OpenAI	ChatGPT-User	Real-time browsing
ClaudeBot	Anthropic	ClaudeBot	Model training
PerplexityBot	Perplexity	PerplexityBot	Search index + training
Bytespider	ByteDance	Bytespider	Model training
Google-Extended	Google	Google-Extended	Gemini training (NOT search)
CCBot	Common Crawl	CCBot	Open dataset

Key distinctions:

• Blocking Google-Extended prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use Googlebot)
• Blocking GPTBot prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing ()