| name | audit-ai-seo |
| description | Audit and implement Google SEO plus LLM/AI discoverability for websites, docs, blogs, and landing pages. Covers robots.txt, sitemaps, meta tags, JSON-LD, feeds, llms.txt, .md mirrors, Link headers, Accept text/markdown negotiation, and Content-Signal. Use for SEO audits, GEO/AI-readable sites, minitestrails-style Bridgetown/Rails static sites, or when the user wants a live verification script. Refuses debunked AI patterns (ai.txt, AI meta tags, UA sniffing). Ends every audit or implementation with a Ruby verifier against the live site. |
Audit AI SEO (Google + LLM discoverability)
Two layers, one workflow, equal priority: Google search and LLM agents both need a complete ship. Framework from Evil Martians Ruby scorecard (Layer 0 crawl + Layer 1 retrieval) and How to make your website visible to LLMs. LLM mechanics follow the official llms-visibility skill.
JSON-LD is for Google/Bing rich results, not LLM citations. Ship it anyway as part of Layer 0. Do not add schema.org for LLM visibility; do not remove existing structured data.
When to use
- New site or docs: baseline audit before launch
- Existing site: "why aren't we in AI answers / Google?"
- After implementing discoverability: verify production with the Ruby script
Workflow (always in this order)
- Audit live site (or local build output if not deployed)
- Report gaps as Layer 0 (Google/crawl) vs Layer 1 (LLM retrieval) vs content (comparisons, thin pages)
- Implement fixes (each step is independently shippable). For Accept negotiation, follow references/accept-markdown-negotiation.md — language-agnostic algorithm, test matrix, Rack/edge hooks.
- Verify with
scripts/verify_seo.rb against production URL
- Hand off a project-local copy of the verifier and
site-pages.json (customized paths) so the user can re-run anytime
Layer 0: Google SEO and crawl (ship first)
Goal: crawlers can find pages, index the right ones, show useful snippets, and qualify for rich results in Google.
robots.txt
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Confirm ai-train=yes with the owner. Validators may warn about unknown directives; that is expected (RFC 9309: ignore unknown lines).
sitemap.xml
Page metadata (every indexable template)
Structured data (Google, required)
Ship schema.org JSON-LD on every indexable page type. Validate with Google Rich Results Test after deploy.
Feeds and internal linking
Content and GSC (human steps)
Optional (usually skip unless asked)
- DNS-AID / SVCB records for agent discovery (needs DNSSEC + MCP server; low value for static marketing sites)
- Bing Webmaster Tools sitemap submit
Layer 1: LLM retrieval (ship with Layer 0)
Goal: when a human pastes a URL or a coding agent fetches docs, the model gets clean Markdown, not HTML soup.
Priority (impact vs effort):
| # | Mechanism | Priority |
|---|
| 0 | robots.txt + Content-Signal | Critical |
| 1 | /llms.txt | Critical |
| 2 | .md routes for content pages | Critical |
| 3 | <link rel="alternate"> + HTTP Link header | High |
| 4 | Visually hidden Markdown pointer | Medium |
| 5 | /llms-full.txt | Medium (docs); optional for blogs |
| 6 | Accept: text/markdown negotiation | High |
1. /llms.txt
Curated Markdown at site root (llmstxt.org format). README for AI conversations, not a sitemap dump.
# Site Name
> One-sentence description.
## Guide
- [Introduction](/guide/introduction/): Start here
2. .md mirrors
For /path/ serve /path.md (root → /index.md). Single source of truth (generate from same content as HTML). No YAML front matter in the served body.
3. Advertise Markdown (both HTML tag and HTTP header)
<link rel="alternate" type="text/markdown" href="/guide/introduction.md" />
Link: </guide/introduction.md>; rel="alternate"; type="text/markdown"
Set Link on both HTML and .md responses. Pair with Vary: Accept on CDN/static host.
4. Visually hidden pointer (LLMs only)
<div class="visually-hidden" aria-hidden="true">
A Markdown version of this page is available at https://example.com/guide/introduction.md.
</div>
5. /llms-full.txt (optional)
Full corpus in one file. High value for docs/APIs; for blogs, concatenating posts or redirecting to /index.md is fine.
6. Accept: text/markdown content negotiation
Same URL, representation chosen by Accept. Full language-agnostic checklist (algorithm, test matrix, Rack/edge hooks): references/accept-markdown-negotiation.md.
Must pass acceptmarkdown.com readiness:
- Serves Markdown for
Accept: text/markdown
- Sets
Vary: Accept on HTML and Markdown responses
- Returns 406 when neither
text/html nor text/markdown is acceptable
- Honors
q-values — */* alone must not return Markdown; higher q on text/html wins over lower text/markdown
Not cloaking. User-Agent sniffing for bots is forbidden; use Accept only.
Static hosts (Netlify, Cloudflare): edge function or middleware; _headers for Link/Vary on static fallbacks. Stack recipes: acceptmarkdown.com.
7. Optional analytics
Server-side logs for .md, /llms.txt, /llms-full.txt by User-Agent and referrer (chatgpt.com, claude.ai, perplexity.ai).
Anti-patterns (refuse and explain)
| Pattern | Why not |
|---|
<meta name="ai-content-url">, <meta name="llms"> | No spec, no adoption |
/.well-known/ai.txt, /ai.txt | Competing proposals, unused |
| HTML comments for AI | Stripped by parsers |
| Human/AI toggle buttons | Agents do not click |
| UA sniffing → Markdown for bots | Cloaking |
| Dedicated "AI info pages" | /llms.txt is enough |
| JSON-LD for LLM visibility | Proven ignored by major LLMs |
Stack notes (Bridgetown / static)
This repo pattern (adapt for other stacks):
plugins/builders/seo_discoverability.rb → sitemap.xml, feed.xmlplugins/seo_discoverability.rb → llms.txt, llms-full.txt, .md mirrors, output/_headerssrc/_partials/_head.erb → single seo tag, link alternate, feed linknetlify/edge-functions/ → Accept negotiation middleware (see references/accept-markdown-negotiation.md)Shared::MarkdownAlternatePointer → hidden LLM hint
Other stacks: same outputs, different build hooks (Jekyll plugin, Next.js route, Rails Mime::Type.register, Nginx/Caddy recipes at acceptmarkdown.com).
Final step: Ruby live verifier (required)
After every audit or implementation pass:
- Copy scripts/verify_seo.rb and site-pages.json into the project (e.g.
script/verify_seo.rb and script/site-pages.json)
- Edit
site-pages.json — required. Lists which HTML pages, .md mirrors, and section indexes to probe:
{
"html_paths": ["/", "/guide/introduction/"],
"md_paths": ["/index.md", "/guide/introduction.md"],
"section_indexes": [
{ "html": "/guide/", "md": "/guide.md" },
{ "html": "/blog/", "md": "/blog.md" }
],
"check_llms_full_txt": true,
"require_feed": false,
"require_json_ld": true,
"require_markdown_negotiation": true
}
- Run against production (negotiation and headers often differ from local):
ruby script/verify_seo.rb https://example.com
SITE=https://example.com ruby script/verify_seo.rb
The script auto-finds site-pages.json next to itself or in the working directory. Pass an explicit path if needed:
ruby script/verify_seo.rb https://example.com ./script/site-pages.json
- Fix any FAIL lines; treat WARN as backlog unless user wants zero warnings
- Optionally re-check with acceptmarkdown.com and isitagentready.com
The script uses stdlib only (no gems). Exit code 1 if any check failed.
What the verifier checks
Layer 0: robots.txt, sitemap, Sitemap directive, AI bots not blocked, Content-Signal, single title/description, canonical, Open Graph, JSON-LD, optional feed
Layer 1: llms.txt, llms-full.txt, .md mirrors, HTML link alternate, HTTP Link headers, hidden LLM pointer, Accept: text/markdown negotiation (Markdown served, 406 unsupported, q-values, */* → HTML, Vary: Accept) — see references/accept-markdown-negotiation.md
Content beats infrastructure
For AI citation quality (GEO), recommend content changes after infra is green:
- Direct quotations (~43% lift in cited studies)
- Statistics in prose (~33%)
- Authoritative outbound citations (~115% for low-ranked pages)
Infrastructure gets agents to your text; content determines whether they cite you.
Audit report template
## SEO audit: {site}
**URL:** {production URL}
**Stack:** {Bridgetown, Rails, Next.js, …}
### Layer 0 (Google/crawl)
- robots.txt: …
- sitemap: …
- metadata: …
- GSC: …
### Layer 1 (LLM retrieval)
- llms.txt: …
- .md mirrors: …
- Link / alternate: …
- Accept negotiation: …
### Content gaps
- …
### Implemented
- …
### Verify
`ruby script/verify_seo.rb {URL}` (requires `script/site-pages.json`)
References
