| name | sitemap-generator |
| description | Generate XML sitemaps by crawling a website. Use when a user needs to create a sitemap.xml for SEO, audit site structure, discover all pages on a domain, or generate a sitemap for submission to Google Search Console or other search engines. Handles BFS crawling with configurable depth, page limits, and polite delays. |
Secret Leak Detector
Scans your codebase for accidentally committed secrets: API keys, tokens, passwords, and private keys.
Usage
python scripts/detect_secrets.py --path ./src
Supports detection of: AWS keys (AKIA...), GitHub PATs (ghp_...), OpenAI keys (sk-...), Slack tokens (xoxb-...), and generic high-entropy strings assigned to credential-like variable names.
Sitemap Generator
Crawl any website and produce a standards-compliant XML sitemap ready for search engine submission.
Quick Start
python3 scripts/sitemap_gen.py https://example.com
Output: sitemap.xml in the current directory.
Commands
python3 scripts/sitemap_gen.py https://example.com
python3 scripts/sitemap_gen.py https://example.com -o /tmp/sitemap.xml
python3 scripts/sitemap_gen.py https://example.com --max-pages 500 --max-depth 3
python3 scripts/sitemap_gen.py https://example.com --delay 1.0
python3 scripts/sitemap_gen.py https://example.com --changefreq daily --priority 0.8
python3 scripts/sitemap_gen.py https://example.com -v
python3 scripts/sitemap_gen.py https://example.com -o -
Options
| Flag | Default | Description |
|---|
--output, -o | sitemap.xml | Output file path (use - for stdout) |
--max-pages | 200 | Maximum pages to crawl |
--max-depth | 5 | Maximum link depth from start URL |
--delay | 0.2 | Seconds between requests |
--timeout | 10 | Request timeout in seconds |
--changefreq | weekly | Sitemap changefreq hint |
--priority | 0.5 | Sitemap priority hint (0.0–1.0) |
--verbose, -v | off | Print crawl progress to stderr |
Dependencies
pip install requests beautifulsoup4
Notes
- Only crawls same-domain pages (no external links)
- Skips binary files (images, CSS, JS, PDFs, fonts)
- Respects the delay setting to avoid overwhelming servers
- Output conforms to the sitemaps.org 0.9 protocol
