robots-txt-discipline

name	robots-txt-discipline
description	Use when controlling crawler access. Robots.txt specifies what crawlers can/can't access.

Robots.txt Discipline

Robots.txt = crawler instructions. Block: /admin, /api, /staging, /private. Allow: everything else by default. Don't block JS/CSS.

Robots.txt format

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /staging/

Sitemap: https://cardprepai.com/sitemap.xml

Common mistakes

1. Blocking /css/ or /js/ — Google can't render the page properly
2. Blocking important pages thinking it boosts SEO (it doesn't)
3. Forgetting to update robots.txt after launching
4. Blocking /search/ but having useful search-result pages
5. Block all in development; forget to remove in production

Testing robots.txt

1. https://cardprepai.com/robots.txt — verify accessible
2. Search Console > Robots.txt Tester (validates rules)
3. Test individual URL: 'Is /admin blocked?' Yes/No
4. View Source-of-truth — should match production

Where this fits in the X3 empire

Robots.txt rule for every CardPrepAI deployment.