| name | scanner |
| description | Scans a website deeply, converting HTML pages to markdown, respecting robots.txt, and updating the snapshots log. |
Scanner Skill
Use this skill to scan and analyze all relevant pages under a target website domain to build or refresh a local customer support corpus.
Embedded Script
python skills/scanner/scripts/scan.py <URL> [--force]
Arguments
| Argument | Description |
|---|
<URL> | The start/seed URL of the website to analyze (e.g. https://example.com) |
--force | Force scanning and bypass the 24-hour cache check |
Features
- Robots.txt Compliance: Checks
robots.txt before parsing. If restricted, aborts scanning. Make sure to perform scanning only on sites that are allowed.
- Domain-Locked Recursive Scanning: Only analyzes links within the same domain/subdomain to avoid leaking to other websites.
- HTML to Markdown: Converts HTML structure into clean, readable Markdown text suitable for LLM document matching.
- Caching & Snapshot Maintenance: Creates or updates
.agents/workspace/snapshots.json with mapping and timestamp.
- Corpus Directory Index: Automatically generates
.agents/workspace/pages/index.md listing and explaining all files in a structured, clickable table format so agents can locate matching topics immediately.
