| name | llms-txt-checker |
| description | Audits any domain's AI-readiness by using curl to directly probe robots.txt, llms.txt, and llms-full.txt, then scores each file against a structured checklist and delivers a formatted report with pass/warn/fail findings and actionable fixes. Use this skill whenever a user provides a domain or URL and wants to know if llms.txt or llms-full.txt is available, discoverable, or properly structured. Trigger on phrases like "check llms.txt for", "does this site have llms.txt", "find llms.txt", "check llms for this url", "audit llms.txt", "is llms-full.txt available", or any time a user shares a domain/docs URL and wants AI-readiness checked. Also trigger when the user wants to verify GEO/AEO readiness of a documentation site.
|
LLMs.txt Checker Skill
Audits any domain's AI-readiness by using curl to directly probe robots.txt, llms.txt, and llms-full.txt, then scores each file against a structured checklist and delivers a formatted report with pass/warn/fail findings and actionable fixes.
The user provides only a domain (e.g. anthropic.com or docs.example.com). Claude uses bash_tool with curl commands to directly probe the domain — no guessing, no page-scraping required.
How it works
Instead of relying on web_fetch and hoping links surface organically, this skill uses curl via bash_tool to directly request the well-known paths for robots.txt, llms.txt, and llms-full.txt. This is reliable, fast, and works regardless of how the site is built.
The curl commands follow HTTP redirects, capture response codes, and save content to temp files for auditing.
Step-by-Step Workflow
Step 1: Normalise the domain
Take the user-provided input and strip any trailing slashes, http://, https://, or path segments to get a clean base domain (e.g. docs.anthropic.com). If the user provides a full URL like https://docs.anthropic.com/en/home, extract just docs.anthropic.com.
Step 2: Fetch all three files via curl
Run the following curl commands using bash_tool. Use -L to follow redirects, -s for silent mode, -o to save content, -w to capture HTTP status codes, and a reasonable timeout (--max-time 10).
DOMAIN="<normalised-domain>"
curl -L -s -o /tmp/robots.txt -w "%{http_code}" --max-time 10 "https://$DOMAIN/robots.txt" > /tmp/robots_status.txt
curl -L -s -o /tmp/llms.txt -w "%{http_code}" --max-time 10 "https://$DOMAIN/llms.txt" > /tmp/llms_status.txt
curl -L -s -o /tmp/llms-full.txt -w "%{http_code}" --max-time 10 "https://$DOMAIN/llms-full.txt" > /tmp/llms_full_status.txt
echo "robots.txt: $(cat /tmp/robots_status.txt) | $(wc -c < /tmp/robots.txt) bytes"
echo "llms.txt: $(cat /tmp/llms_status.txt) | $(wc -c < /tmp/llms.txt) bytes"
echo "llms-full.txt: $(cat /tmp/llms_full_status.txt) | $(wc -c < /tmp/llms-full.txt) bytes"
Interpret the HTTP status codes:
- 200 → file exists, read and audit the content
- 301/302 → followed automatically by
-L; final destination counts
- 404 → file does not exist at this path
- 403/429/5xx → server-side block or error; note it explicitly
- 000 → connection failed (domain unreachable or timeout)
Step 3: Read and classify results
After the curl commands complete, read the saved files:
cat /tmp/robots.txt
cat /tmp/llms.txt
head -200 /tmp/llms-full.txt
wc -l /tmp/llms-full.txt
wc -c /tmp/llms-full.txt
Case A — Both llms.txt (200) AND llms-full.txt (200)
- Both files fetched successfully; proceed to the Audit Checklist (Step 4)
Case B — Only llms.txt (200), llms-full.txt returned 404
- Audit
llms.txt
- Scan its content for any internal reference to
llms-full.txt (it may be hosted at a non-standard path)
- If a custom path is found → curl that path and audit it
- If not found → report
llms-full.txt as absent and not referenced
Case C — llms.txt returned 404
- Report that neither file is present at the standard paths
- Note whether
robots.txt gave any hints (some sites reference llms.txt inside robots.txt)
- Report clearly to the user (see Response Templates section below)
robots.txt (always check regardless of Case)
- Even if
llms.txt is missing, always read and audit robots.txt for AI-access signals
Step 4: Audit the files
llms.txt Audit
Check for the following. Mark each ✅ or ❌:
Structure
Content completeness
AI-readiness signals
llms-full.txt Audit (if available)
robots.txt Signal (check opportunistically)
If robots.txt was surfaced during the process:
Step 5: Deliver the report
Structure the output as:
## LLMs.txt Audit: [domain]
### Discovery
[What was found and how it was surfaced]
### llms.txt — ✅ Found / ❌ Not Found
[Audit results with ✅/❌ per checklist item]
[Notable strengths]
[Issues found]
### llms-full.txt — ✅ Found / ❌ Not Found / ⚠️ Not Referenced
[Audit results or explanation]
### robots.txt Signal
[If available — what it says about AI access]
### Summary & Recommendations
[3–5 actionable bullets]
Response Templates
Neither llms.txt nor llms-full.txt surfaced
Neither llms.txt nor llms-full.txt was discoverable from the provided URL.
This means AI agents and LLMs browsing your docs will have no structured index to work from — they'll need to crawl individual pages or guess at your content structure.
To fix this, surface the llms.txt URL somewhere Claude (and other AI tools) can see it when fetching your page. Good options:
- Add it to your page footer (e.g.
LLM usage: /llms.txt)
- Include it in a blockquote at the top of your docs homepage or
.md page version (e.g. > Documentation index available at: https://yourdomain.com/llms.txt)
- Reference it in your
robots.txt or a <meta> tag
Once it's linked from a page that AI agents naturally land on, it becomes discoverable automatically.
llms.txt found but llms-full.txt not referenced
llms.txt was found and audited. However, llms-full.txt was not referenced anywhere in the file.
llms-full.txt is the companion file containing the full content of all documentation pages in a single file — useful for AI coding assistants (Cursor, Claude Code, Copilot) that need deep context without fetching dozens of individual pages.
To add it: Reference it in your llms.txt under a ## Documentation Sets section or similar, like:
- [Complete documentation](https://yourdomain.com/llms-full.txt): full content of all pages
If you're on Mintlify, it's auto-generated — just make sure it's linked.
Key facts to keep in mind
- Mintlify auto-generates both
llms.txt and llms-full.txt for all projects, and adds HTTP headers (Link: </llms.txt>; rel="llms-txt") for discovery
- Fern also auto-generates both files
- Starlight (Astro) does not auto-generate — must be added manually
- GitBook auto-generates
llms.txt
- The
llms.txt standard was proposed by Jeremy Howard (fast.ai) in September 2024
llms-full.txt is not part of the original spec but has become widely adopted as the companion file- No major AI crawler has officially committed to following these files, but Cursor, Claude Code, and similar tools actively use them
