| name | marketing-ai-crawler |
| description | Report which AI bots (GPTBot, ClaudeBot, Claude-User, PerplexityBot, Bytespider, CCBot, GoogleOther, and more) are crawling your Vercel-hosted sites, and which paths they hit. Reads Vercel request logs (with user agents) and prints a per-site bot -> hits -> top paths table. Triggers on "which AI bots crawl", "ai crawler report", "is GPTBot hitting", "is ClaudeBot crawling", "AI bot traffic", "who's crawling the site", or "crawler report for [site]". |
AI Crawler Report (Vercel)
Per-site table of AI crawler activity — bot → hits → top paths — for Vercel-hosted
sites, pulled from Vercel request logs (which, unlike vercel logs CLI output,
include the client user agent).
Setup — you do 1 thing, I do the rest
- You: be logged in to the Vercel CLI (
vercel login) on this machine — the
skill reuses that token (or export VERCEL_TOKEN).
Then ask me to run setup + doctor; I resolve the team and project IDs and write
the machine-local config (~/.config/ai-crawler-report/sites.json — never in
the repo).
cd <skill-dir>
python3 cli.py setup --team <team-slug> --projects drafty=drafty.im,myapp=myapp.com
python3 cli.py doctor
No dependencies — stdlib Python only.
Commands
cd <skill-dir>
python3 cli.py report
python3 cli.py report --since 6h --site myapp.com --json
Output per site:
myapp.com (4,213 requests scanned)
──────────────────────────────────────────────
Bot Hits Top paths
GPTBot 142 /retatrutide-dosage (38), / (22), ...
PerplexityBot 31 / (12), /faq (8), ...
Every run also saves a JSON snapshot to <skill-dir>/results/ so runs can be
compared over time.
Coverage — be honest about the window
- Data path (verified 2026-06): Vercel Observability has bot/crawler
insights, but they are dashboard-only on regular plans — the query builder
and CSV/JSON export need Observability Plus, and there is no public bot API.
This skill instead reads the request-logs backfill endpoint that powers
vercel logs (vercel.com/api/logs/request-logs), authenticated with the
CLI's own token. That endpoint returns clientUserAgent per request, which
the CLI drops. It is internal — if it changes, run doctor and check here.
Its page param is ignored server-side, so cli.py paginates by
time-slicing endDate and deduping on requestId.
- Retention (verified 2026-06): on regular plans the backfill only goes
back ~1 day; asking for ≥3 days returns
ExceedsBillingLimitError. So a
weekly run reports the trailing ~24h, not the whole week — the report says
so in its footer. For trends, run daily (cron/schedule) and compare the
saved snapshots in results/.
- Static/CDN-served requests ARE included (the endpoint logs all requests,
not just function invocations).
Bot list notes
The matcher covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User,
Claude-SearchBot, PerplexityBot, Perplexity-User, GoogleOther, Bytespider,
CCBot, Meta-ExternalAgent, Amazonbot, Cohere, DuckAssistBot, MistralAI-User,
YouBot, AI2Bot, Diffbot, Timpibot (UA substrings in cli.py, verified
2026-06). Google-Extended and Applebot-Extended are robots.txt tokens, not
user agents — they never appear in logs; Gemini-related crawling shows up as
GoogleOther / Google-CloudVertexBot.
Troubleshooting
doctor fails on token → vercel login, or export VERCEL_TOKEN.ExceedsBillingLimitError → shrink --since (plan retention cap).- A site shows 0 requests scanned → check the project name in
~/.config/ai-crawler-report/sites.json (re-run setup).
- Truncation warning → the safety cap (20k requests/site) hit; use a smaller
--since.
