菜单
Use when auditing robots.txt, AI crawler permissions, GEO crawl policy, cornerstone path crawlability, or blocking/allowing GPTBot, PerplexityBot, OAI-SearchBot, and Google-Extended. Audits selective crawler rules per bot (training vs indexing crawl), validates Sitemap directives, and drafts policy-aligned robots.txt.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Use when analyzing fan-out coverage, sub-query coverage, topic coverage gaps, or keyword-only SERP fan-out without a domain. Simulates AI query fan-out and optional domain coverage checks for a keyword.
Use when creating or reviewing an agent skill, adapting an external skill for public release, reviewing SKILL.md quality, or choosing handoff JSON and verify scripts. Right-sized packaging with predictable process; regression gates only when needed.
Use when auditing keyword cannibalization, same-domain page overlap, or hub-vs-hub conflicts before consolidation. Use for accidental hub vs intended hub, blog post outranking a landing page, or internal competition diluting rankings on one domain.
Use when recovering hub-and-spoke content architecture on a messy site — accidental hubs, hidden gems, legacy bloat, or onboarding a client without a designed pillar map. Use for organic architecture discovery, de facto pillar audit, GSC and GA4 triangulation, or bottom-up cornerstone recovery. Triggers: "hub and spoke audit", "discover my real pillars", "accidental cornerstone", "hidden gem pages", "performance matrix content audit", "triangulate GSC and GA4 with keyword data".
Use when running a cornerstone content audit, pillar content review, or content architecture audit. End-to-end: discover pages, identify cornerstones, health audit, cannibalization check, and enhancement plan.
Use when triaging landing pages with Google Analytics (GA4) and Google Search Console exports — query intent and landing-page discovery, optional conversion quadrants, accidental hubs, or prune review before hub-and-spoke recovery. Ask whether the audit should be discovery-only or discovery-plus-conversions. Works in GA4-only or GSC-only partial mode when one source is missing. Domain-agnostic; CSV or API-shaped tables.
| name | robots.txt-Audit |
| type | capability |
| description | Use when auditing robots.txt, AI crawler permissions, GEO crawl policy, cornerstone path crawlability, or blocking/allowing GPTBot, PerplexityBot, OAI-SearchBot, and Google-Extended. Audits selective crawler rules per bot (training vs indexing crawl), validates Sitemap directives, and drafts policy-aligned robots.txt. |
| version | 1.7.1 |
Audit /robots.txt as the authoritative crawl-permission file for search and AI crawlers. Distinguish indexing/answer crawl from training crawl per bot; validate Sitemap:; recommend or draft policy-aligned rules.
Not this skill: deploying files, WAF/CDN config, log analysis, or llms.txt as crawl control.
Docs: Public guide · REFERENCE.md (rubric R1–R11, handoff schema, templates) · REQUIREMENTS.md · EXAMPLES.md · LAYPERSON-OUTPUT.md (chat contract) · RECHECK.md · AGENTS.md
SSOT scripts: run-audit.mjs (primary) · assess-policy.mjs · build-handoff.mjs · build-layperson-summary.mjs · parse-robots-txt.mjs · syntax-validation.mjs · crawler-registry.mjs
Cloudflare: CLOUDFLARE-MANAGED.md when managed markers detected.
max_discovery | block_training_allow_answers | restrictive | audit_only (default audit_only). max_discovery is discovery-only — training blocks are optional.robots_deployment (optional) — auto | origin_only | cloudflare_managedIf domain missing, ask once. Never infer domain from examples.
| Mode | When |
|---|---|
| audit | File found, compliant or audit_only |
| recommend | Policy gaps vs crawl_policy |
| recheck | User updated file — see RECHECK.md |
| generate | File missing or user requests draft |
https://{domain}/robots.txt and https://www.{domain}/robots.txt.[{ "url", "status" }].index_control_results.cd marketing/skills/robots-txt-audit
node scripts/run-audit.mjs \
--robots path/to/fetched.txt \
--domain example.com \
--policy max_discovery \
--sitemap-fetch path/to/sitemap-fetches.json
PowerShell (use ; instead of &&):
Set-Location marketing/skills/robots-txt-audit; node scripts/run-audit.mjs --robots path/to/fetched.txt --domain example.com --policy audit_only --sitemap-fetch path/to/sitemap-fetches.json
audit_checks or handoff JSON — build-handoff.mjs produces G-valid handoff inside the detail file.Recheck: add --recheck (reads reports/{domain}-latest-snapshot.json). See RECHECK.md.
Forbidden in chat: per-bot matrix, rubric tables, handoff JSON, MD_* IDs. See LAYPERSON-OUTPUT.md.
discovery.urls_checked; set discovery.resolved_url.discovery.fetch_suspect: true to assess.Both markers present → read CLOUDFLARE-MANAGED.md full; deployment.model: cloudflare_managed; use layer split. Do not duplicate CF training blocks at origin.
Run assessRobotsTxtContent(content, crawl_policy, domain, options) with:
sitemap_fetch_results — required when sitemaps declaredindex_control_results — when cart/checkout probedsyntax_validation from output — do not re-lint in proseRubric, per-bot matrix columns, R7 sub-checks, policy presets, and audit_findings tiers: REFERENCE.md (not repeated here).
Templates and draft rules: REFERENCE.md. generate drafts must pass policy_compliance for the chosen preset.
Full schema and field rules: REFERENCE.md. crawler_matrix must be a JSON array from assessment (never a string placeholder — fails G26).
www in urls_checked (G27)run-audit.mjs or equivalent writeAuditReports with fetch options forwardedLAYPERSON-OUTPUT.md)reports/{domain}-{date}-detail.mdRECHECK.md; snapshot updatedCLOUDFLARE-MANAGED.md; G28–G33 fields when applicableShip bar:
cd marketing/skills/robots-txt-audit
node scripts/verify-all.mjs