| name | agent-ready-robots-txt |
| description | Sub-skill de agent-ready-cloudflare: Implement robots.txt
|
Implement robots.txt
Publish a valid robots.txt at your site root per
RFC 9309.
Requirements
- Serve
/robots.txt as text/plain with HTTP 200
- Include
User-agent directives with Allow/Disallow rules
- Reference your sitemap if one exists:
Sitemap: https://example.com/sitemap.xml
Cloudflare
AI Crawl Control
can manage your robots.txt from the dashboard, including AI-specific bot rules.
Validate
POST https://isitagentready.com/api/scan
Content-Type: application/json
{"url": "https://YOUR-SITE.com"}
Check that checks.discoverability.robotsTxt.status is "pass".