メニュー
Crawl and sanitize external websites for LLM ingestion. Use when needing to crawl comindware.ru or cmwlab.com, sanitize scraped content, or manage the crawl→sanitize pipeline. Supports --fresh (from scratch) and --resume (checkpoint) workflows.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
Use when you've completed a non-trivial task and need to document discoveries, patterns, or gotchas for future sessions. Also use to review the discovery log before starting similar work.
Transcribe meeting recordings with Google Gemini for future documentation work. Use when the user provides a video file and needs a structured Markdown transcript with timestamps, visual notes, and summary in .scratch/.
Use when working on the PHPKB cloning and post-clone migration workflow in this repository: creating a new PHPKB section for a new product version, publishing a new MkDocs article by cloning an adjacent PHPKB article, syncing changed for_kb_import_ru HTML back to PHPKB by kb-id (git diff batch), cloning PHPKB categories or articles, updating cloned PHPKB article/category links, migrating local docs IDs with clone mappings, fixing related topics after cloning, or analyzing/updating scripts in utilities/phpkb_cloning.
Use when writing or answering about N3/Turtle/RDF/Notation3/Triples in Comindware Platform
Reference files and patterns for writing C# scripts in Comindware Platform
Use when regenerating or ingesting the Comindware Platform LLM knowledge bundle in this repository: refreshing phpkb_content_rag from PHPKB, running phpkb_import_for_rag.py, running phpkb_ingest.py, or updating the LLM ingestion bundle.
| name | scrape-sanitize |
| description | Crawl and sanitize external websites for LLM ingestion. Use when needing to crawl comindware.ru or cmwlab.com, sanitize scraped content, or manage the crawl→sanitize pipeline. Supports --fresh (from scratch) and --resume (checkpoint) workflows. |
Crawl external websites and sanitize the output for LLM ingestion.
scraping/{site}/
├── progress_{date}.json # Crawl progress (tracked, for resumability)
├── sanitize_checkpoint.json # Sanitizer checkpoint (tracked, for resumability)
└── sanitized_{date}.md # Final sanitized output (tracked)
.scratch/
└── {site}_dirty_{date}.md # Raw crawl output (gitignored, disposable)
| Script | Purpose |
|---|---|
scripts/crawl4ai_ingest.py | Async crawl via crawl4ai (sitemap → markdown) |
scripts/http_bs4_ingest.py | Sync crawl via requests+BS4 (markdownify conversion) |
scripts/sanitize.py | Boilerplate pruning, URL decoding, title resolution |
python .agents/skills/scrape-sanitize/scripts/crawl4ai_ingest.py --site comindware_ru
python .agents/skills/scrape-sanitize/scripts/crawl4ai_ingest.py --site comindware_ru --fresh # start from scratch
python .agents/skills/scrape-sanitize/scripts/sanitize.py --site comindware_ru --date 20260616
python .agents/skills/scrape-sanitize/scripts/sanitize.py --site comindware_ru --date 20260616 --fresh
python scripts/crawl4ai_ingest.py --site comindware_ru --fresh
python scripts/sanitize.py --site comindware_ru --date 20260616 --fresh
scraping/{site}/progress_{date}.json tracks processed URLs. Resume with --resume (default). Clear with --fresh.scraping/{site}/sanitize_checkpoint.json tracks processed articles. Resume with --resume (default). Clear with --fresh.| Site key | Target | Crawler |
|---|---|---|
comindware_ru | https://www.comindware.ru | crawl4ai |
cmwlab_com | https://kb.cmwlab.com | http_bs4 |