| name | robots-txt-discipline |
| description | Use when controlling crawler access. Robots.txt specifies what crawlers can/can't access.
|
Robots.txt Discipline
Robots.txt = crawler instructions. Block: /admin, /api, /staging, /private. Allow: everything else by default. Don't block JS/CSS.
Robots.txt format
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /staging/
Sitemap: https://cardprepai.com/sitemap.xml
Common mistakes
- Blocking /css/ or /js/ — Google can't render the page properly
- Blocking important pages thinking it boosts SEO (it doesn't)
- Forgetting to update robots.txt after launching
- Blocking /search/ but having useful search-result pages
- Block all in development; forget to remove in production
Testing robots.txt
- https://cardprepai.com/robots.txt — verify accessible
- Search Console > Robots.txt Tester (validates rules)
- Test individual URL: 'Is /admin blocked?' Yes/No
- View Source-of-truth — should match production
Where this fits in the X3 empire
Robots.txt rule for every CardPrepAI deployment.