site-crawler-stinger

# Site Crawler Stinger You are equipping **site-crawler-wasp-drone**, part of the Website Auditor by Legion Code Inc. plugin. Full scope and acceptance criteria: [prd-007-site-crawler](../../library/requirements/backlog/prd-007-site-crawler/prd-007-site-crawler-index.md). ## Purpose Runs a platform-aware crawl of the audited site, up to 100 pages, storing raw HTML and a Markdown extraction per page under `site-data/<slug>.html` / `<slug>.md`, plus a `site-data/manifest.json` index. This Stinger's output is the shared, write-once data layer that every one of the nine Wave-5 audit Drones (technical-seo, aeo-audit, content-semantics, internal-linking, visual-funnel, accessibility-audit, web-security-posture, analytics-stack, performance-cwv) reads read-only, with zero write contention between them, per PRD-007's shared workspace contract. ## When to use - Wave W4, sync, immediately after `stack-fingerprint-wasp-drone` has written `_shared/target-profile.json` (this Drone's stack-type dependency, per the build plan's dependency graph and PRD-007's `Depends on: prd-003`) - Any re-crawl after a mid-engagement site restructure is suspected (full re-crawl, not a patch, see `guides/05-politeness-and-scope-limits.md`) - Never invoke this Stinger for a single-page check; that is `stack-fingerprint-stinger`'s scope ## When not to use - Not for classifying the site's technology stack. That is `stack-fingerprint-stinger`, which runs first and writes the `target-profile.json` this Stinger reads. - Not for analyzing what was crawled (SEO, AEO, accessibility, security, performance, content, linking, funnel, or analytics findings). This Stinger only fetches and stores; every Wave-5 Drone owns its own analysis of `site-data/`. - Not for crawling authenticated or gated areas, submitting forms, or exceeding 100 pages without an explicit user-approved re-run. See PRD-007's Non-Goals. ## Procedure 1. Confirm `_shared/target-profile.json` exists with non-null `platform`, `rendering`, and `confidence`. If missing, stop and hand back to the orchestrating agent. 2. Read the platform value and select the seed-path strategy for it (`guides/02-platform-traversal-strategies.md`). 3. Run `shared/scripts/crawl-extract.py` (or its algorithm) to crawl up to 100 same-domain pages, breadth-first, respecting `robots.txt` and rate limits (`guides/01-crawl-procedure.md`, `guides/05-politeness-and-scope-limits.md`). 4. Store each page as `site-data/<slug>.html` + `site-data/<slug>.md` using the deterministic slugify algorithm, and write `site-data/manifest.json` as the single index (`guides/04-storage-and-manifest-convention.md`). 5. Record every unreachable URL in the manifest's `unreachable[]` array with a reason; never retry into a block (`guides/03-fetching-and-rendering.md`). 6. Write a short run summary into this engagement's `_shared/run-ledger.json` entry for this Drone (pages fetched, pages unreachable, platform strategy used). Do not write into this plugin repository's own `library/`. ## References map - `references/templates/manifest-schema.md`, load when building or consuming `manifest.json` and the exact field contract is needed. - `references/templates/site-data-manifest.example.json`, load as a worked example of the manifest shape before writing or parsing one. - `references/research/distilled-site-crawler.md`, load when a fetching/rendering/platform- detection claim needs verification or a dispute needs settling. - Source research archive (not shipped with the plugin): the shipped distillation summarizes it; do not require it. - `references/scripts/README.md` and `shared/scripts/crawl-extract.py`, load/run to actually execute the crawl. - `guides/01-crawl-procedure.md` through `guides/05-politeness-and-scope-limits.md`, load per-verb as described in each guide's own heading. ## Related drones and stingers - [stack-fingerprint-stinger](../stack-fingerprint-stinger) - runs before this Stinger in wave W1a, writes `_shared/target-profile.json`, this Stinger's binding platform-classification input. - [technical-seo-stinger](../technical-seo-stinger) - one of nine Wave-5 Drones reading `site-data/` read-only; consult when a manifest or storage-convention question is actually a technical-SEO scope question instead. - [aeo-audit-stinger](../aeo-audit-stinger) - another Wave-5 reader of `site-data/`. - [content-semantics-stinger](../content-semantics-stinger) - another Wave-5 reader of `site-data/`, the primary consumer of the `.md` extraction quality caveats in `guides/03-fetching-and-rendering.md`. - [internal-linking-stinger](../internal-linking-stinger) - another Wave-5 reader of `site-data/`, and the most likely consumer of `manifest.json`'s `unreachable[]` array as evidence. - [visual-funnel-stinger](../visual-funnel-stinger), [accessibility-audit-stinger](../accessibility-audit-stinger), [web-security-posture-stinger](../web-security-posture-stinger), [analytics-stack-stinger](../analytics-stack-stinger), [performance-cwv-stinger](../performance-cwv-stinger) - the remaining four of the nine Wave-5 Drones reading `site-data/` read-only. - [keyword-intelligence-stinger](../keyword-intelligence-stinger) - runs one wave earlier (W3, not W4) and does NOT read `site-data/`, since it has not been written yet at that point in the sequence; see that Stinger's `guides/03-tier-3-ai-inference.md` for how it independently sources content instead. ## Critical Directive - You must read all files and context contained within your skill. - In the event your core knowledge does not provide sufficient guidance you must make every attempt to search the internet, related knowledge base documentation files, and other available resources to supplement your knowledge prior to proceeding with your task. - Additional related skills can be found here: - [site-crawler-wasp-drone](../../agents/site-crawler-wasp-drone.md) - this Stinger's paired Drone. - [stack-fingerprint-stinger](../stack-fingerprint-stinger) - upstream dependency, writes `target-profile.json`. ## Ship Gate Ship Gate not applicable to this Stinger's own runtime procedure. This Stinger's output (`site-data/`) is written into the external customer's audit workspace (`www.<domain>-audit/site-data/`), not into this plugin repository's own tracked source, so a site-crawler run produces no committable code inside this repo for security-stinger, quality-stinger, or github-repo-health-stinger to gate. The Ship Gate does apply, unmodified, to any change made to this plugin repository's own files, including this SKILL.md, its guides, its references, and `shared/scripts/crawl-extract.py`, before that change is committed and pushed.