Run any Skill in Manus with one click

seo-ai-crawlers

Stars27

Forks1

UpdatedJune 1, 2026 at 18:28

Audit AI crawler access and citability for a page — confirm retrieval/citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are allowed and the Googlebot vs Google-Extended split is correct, classify training vs search/retrieval vs user-fetch user-agents, check the page is server-rendered enough for non-JS AI crawlers, validate llms.txt / llms-full.txt (also covers M21), and generate a choice-gated robots.txt preset. Module M14. Feeds the AI Visibility score.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

Hainrixz

Hainrixz/claude-seo-ai

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

SKILL.md

readonly

name	seo-ai-crawlers
description	Audit AI crawler access and citability for a page — confirm retrieval/citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are allowed and the Googlebot vs Google-Extended split is correct, classify training vs search/retrieval vs user-fetch user-agents, check the page is server-rendered enough for non-JS AI crawlers, validate llms.txt / llms-full.txt (also covers M21), and generate a choice-gated robots.txt preset. Module M14. Feeds the AI Visibility score.
allowed-tools	Read, Grep, Glob, WebFetch, Bash

seo-ai-crawlers (M14)

Controls whether AI search engines can crawl and cite the page, and whether they can read it without JS. The training-vs-search-vs-fetch distinction is everything. Reference: references/ai-crawlers.md.

Audits

Working from the PageSnapshot (rendered_dom if present, else raw_html) plus the site robots.txt:

Citation access: are retrieval/citation bots — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot — actually allowed (not caught by a broad Disallow: / or a wildcard block)? Confirm Googlebot is not blocked and the Googlebot (search) vs Google-Extended (Gemini training control) split is correct.
User-agent classification: bucket every AI agent in robots.txt into training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), search/retrieval (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered fetch (ChatGPT-User, Claude-User, Perplexity-User). Match user-agents case-insensitively; treat the table in references/ai-crawlers.md as a starting set, not exhaustive.
Renderability for non-JS crawlers: pull the M4 (seo-crawl-render) render result — most AI crawlers do not execute JS. If primary content only appears in rendered_dom and is absent from raw_html, flag it as invisible to AI retrieval.
llms.txt / llms-full.txt (also covers M21): presence at the site root, valid Markdown structure (H1 title, summary blockquote, sectioned link lists), and that linked URLs resolve. Follow references/ai-crawlers.md.

Fixes

AUTO (fixable: auto): a citation-friendly robots.txt preset, choice-gated — the user picks allow-citations (default: allow search/retrieval, opt out of training), allow-all, or block-all. Deterministic, additive, verifiable; emitted as a diff for fix.
AUTO (fixable: auto), disclosure-gated and scored 0: llms.txt / llms-full.txt, generated from the site's own structure only on explicit request (fix --category llms), shown as a diff before writing. Additive and deterministic, but never sold as proven ranking value — the disclosure that it is low/uncertain impact is shown every time.
ADVISORY (fixable: advisory): edge/WAF block for bots that ignore robots.txt (e.g. Bytespider) — the tool never writes infra config. Never fabricate sitemap URLs, contact emails, or link targets — ask the user or leave a clearly-marked TODO placeholder.

Verification

node ${CLAUDE_SKILL_DIR}/../../scripts/parse-robots-sitemap.mjs --url <u> (robots_parse) — parses robots.txt, resolves the effective directive for each AI user-agent, and confirms it.
dom_assert against the M4 render result for the renderability check (content present in raw_html).
When the required data tier is unavailable (e.g. robots.txt unfetchable, no M4 render result), status is needs_api — never a false pass.

Findings

Findings conform to schema/finding.schema.json. evidence.observed quotes the page/robots line; verification.reproduce is the runnable command above; expected_impact is banded + confidence-tagged (no naked %). Examples:

M14.citation_bots.blocked — Disallow: / reaches OAI-SearchBot/Claude-SearchBot/PerplexityBot (status fail, severity 4, fixable: auto, axis ai, confidence established).
M14.render.content_js_only — primary content in rendered_dom but absent from raw_html, invisible to non-JS AI crawlers (status warn, severity 4, fixable: advisory, axis ai, confidence directional).
M14.llmstxt.missing — no /llms.txt at site root (status warn, severity 1, fixable: auto, axis ai, confidence speculative, scored 0).

Honesty

Blocking a training bot does NOT block the matching search bot — they are separate user-agents (GPTBot ≠ OAI-SearchBot, ClaudeBot ≠ Claude-SearchBot). Many "block AI" guides get this wrong.
Bytespider and some agents frequently ignore robots.txt; a robots rule is best-effort. Real enforcement needs an edge rule / WAF (advisory).
robots.txt controls crawling, not indexing — to keep a page out, use a noindex meta tag and don't also Disallow it.
llms.txt has only partial vendor support (Anthropic and Perplexity honor it in retrieval; Google does not use it for AI Overviews/AI Mode; OpenAI uncommitted) and is also useful as IDE/coding-agent context. Still low/uncertain — scored 0, never presented as proven ranking value.

More from this repository

same repository

audit

Hainrixz/claude-seo-ai

Audit a website or web codebase for SEO and AI-search (GEO/AEO) — produces two independent 0-100 scores (Search SEO + AI Visibility) plus a prioritized, evidence-backed report. Read-only; never writes files. Use when the user asks to audit, analyze, check, or score a site's SEO, structured data/schema, meta tags, Core Web Vitals, robots.txt, sitemaps, or llms.txt, or how it ranks in Google and is cited by AI engines (ChatGPT, Perplexity, Google AI Overviews, Gemini, Claude).

2026-06-0127

fix

Hainrixz/claude-seo-ai

Opt-in fixer (the /claude-seo-ai:fix command). Applies the safe, deterministic SEO/AI-search fixes from an audit to the user's code — meta viewport/charset/lang, JSON-LD, robots.txt AI directives, hreflang, sitemaps, OG/Twitter cards, image dimensions, canonical, llms.txt. Dry-run preview by default; writes only after explicit per-change confirmation. Runs only when the user invokes it — never auto-triggered.

2026-06-0127

geo

Hainrixz/claude-seo-ai

Analyze and score only a page's AI-search visibility (GEO/AEO) — answer extractability, fact density, AI-crawler access, entity linking, and llms.txt — and report an AI Visibility score with a citability breakdown. Read-only. Use for "will AI engines cite this?", GEO/AEO, or ChatGPT/Perplexity/Google AI Overviews/Gemini optimization.

2026-06-0127

score

Hainrixz/claude-seo-ai

Recompute and display the two scores (Search SEO + AI Visibility) from the most recent audit's findings, without re-crawling. Use to re-show or refresh the scores after an audit, or to score a saved findings JSON file.

2026-06-0127

seo-core-web-vitals

Hainrixz/claude-seo-ai

Audit Core Web Vitals & page performance — measure LCP, INP, and CLS against p75 field thresholds, diagnose render-blocking resources, unoptimized images, and layout-shift sources, and produce prioritized, advisory-only remediation guidance. Module M15. Feeds the Search SEO score (heavily) and the AI Visibility score (minimally).

2026-06-0127

seo-crawl-render

Hainrixz/claude-seo-ai

Fetch a page (or local files) and build the shared PageSnapshot every audit module reads — raw HTML, rendered DOM when a render MCP is available, response headers, status/redirect chain, and site artifacts (robots.txt, sitemaps, llms.txt). Decides whether JavaScript rendering is needed and records the data tier.

2026-06-0127

name	seo-ai-crawlers
description	Audit AI crawler access and citability for a page — confirm retrieval/citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are allowed and the Googlebot vs Google-Extended split is correct, classify training vs search/retrieval vs user-fetch user-agents, check the page is server-rendered enough for non-JS AI crawlers, validate llms.txt / llms-full.txt (also covers M21), and generate a choice-gated robots.txt preset. Module M14. Feeds the AI Visibility score.
allowed-tools	Read, Grep, Glob, WebFetch, Bash

seo-ai-crawlers (M14)

Audits

Working from the PageSnapshot (rendered_dom if present, else raw_html) plus the site robots.txt:

Citation access: are retrieval/citation bots — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot — actually allowed (not caught by a broad Disallow: / or a wildcard block)? Confirm Googlebot is not blocked and the Googlebot (search) vs Google-Extended (Gemini training control) split is correct.
User-agent classification: bucket every AI agent in robots.txt into training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), search/retrieval (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and user-triggered fetch (ChatGPT-User, Claude-User, Perplexity-User). Match user-agents case-insensitively; treat the table in references/ai-crawlers.md as a starting set, not exhaustive.
Renderability for non-JS crawlers: pull the M4 (seo-crawl-render) render result — most AI crawlers do not execute JS. If primary content only appears in rendered_dom and is absent from raw_html, flag it as invisible to AI retrieval.
llms.txt / llms-full.txt (also covers M21): presence at the site root, valid Markdown structure (H1 title, summary blockquote, sectioned link lists), and that linked URLs resolve. Follow references/ai-crawlers.md.

Fixes

AUTO (fixable: auto): a citation-friendly robots.txt preset, choice-gated — the user picks allow-citations (default: allow search/retrieval, opt out of training), allow-all, or block-all. Deterministic, additive, verifiable; emitted as a diff for fix.
AUTO (fixable: auto), disclosure-gated and scored 0: llms.txt / llms-full.txt, generated from the site's own structure only on explicit request (fix --category llms), shown as a diff before writing. Additive and deterministic, but never sold as proven ranking value — the disclosure that it is low/uncertain impact is shown every time.
ADVISORY (fixable: advisory): edge/WAF block for bots that ignore robots.txt (e.g. Bytespider) — the tool never writes infra config. Never fabricate sitemap URLs, contact emails, or link targets — ask the user or leave a clearly-marked TODO placeholder.

Verification

node ${CLAUDE_SKILL_DIR}/../../scripts/parse-robots-sitemap.mjs --url <u> (robots_parse) — parses robots.txt, resolves the effective directive for each AI user-agent, and confirms it.
dom_assert against the M4 render result for the renderability check (content present in raw_html).
When the required data tier is unavailable (e.g. robots.txt unfetchable, no M4 render result), status is needs_api — never a false pass.

Findings

M14.citation_bots.blocked — Disallow: / reaches OAI-SearchBot/Claude-SearchBot/PerplexityBot (status fail, severity 4, fixable: auto, axis ai, confidence established).
M14.render.content_js_only — primary content in rendered_dom but absent from raw_html, invisible to non-JS AI crawlers (status warn, severity 4, fixable: advisory, axis ai, confidence directional).
M14.llmstxt.missing — no /llms.txt at site root (status warn, severity 1, fixable: auto, axis ai, confidence speculative, scored 0).

Honesty

Blocking a training bot does NOT block the matching search bot — they are separate user-agents (GPTBot ≠ OAI-SearchBot, ClaudeBot ≠ Claude-SearchBot). Many "block AI" guides get this wrong.
Bytespider and some agents frequently ignore robots.txt; a robots rule is best-effort. Real enforcement needs an edge rule / WAF (advisory).
robots.txt controls crawling, not indexing — to keep a page out, use a noindex meta tag and don't also Disallow it.
llms.txt has only partial vendor support (Anthropic and Perplexity honor it in retrieval; Google does not use it for AI Overviews/AI Mode; OpenAI uncommitted) and is also useful as IDE/coding-agent context. Still low/uncertain — scored 0, never presented as proven ranking value.