| name | llms-txt-and-crawler-access |
| description | Review crawler and AI-agent access files for ecommerce sites, including robots.txt, sitemap, llms.txt, and AI bot rules. |
LLMs.txt And Crawler Access
Workflow
- Check
robots.txt for blocked product/category/policy paths and AI bot rules.
- Check sitemap availability, freshness, and product/category coverage.
- Check whether
llms.txt exists and points to useful public resources.
- Check canonical URLs and redirects that may confuse crawlers.
- Recommend minimal changes that improve agent discoverability without exposing private areas.
Delegation
This skill is the ecommerce-scoped access review. For depth, delegate to the ai-visibility pack instead of duplicating its work:
- Full robots/meta-robots/header analysis →
robots-ai-crawler-audit.
- Sitemap coverage, canonicals, and redirect chains →
sitemap-discovery-audit.
- Drafting a new
llms.txt → llms-txt-generator.
Guardrails
- Never recommend opening admin, cart, checkout session, account, or private order paths to crawlers.
- Keep access recommendations scoped to public storefront evidence.