// Use when analyzing server logs to understand how AI crawlers (GPTBot, ClaudeBot, PerplexityBot) interact with your site. Use when optimizing content placement for LLM retrieval, diagnosing why AI search isn't citing your content, or auditing crawl patterns to find optimization gaps.
| name | ai-bot-log-audit |
| description | Use when analyzing server logs to understand how AI crawlers (GPTBot, ClaudeBot, PerplexityBot) interact with your site. Use when optimizing content placement for LLM retrieval, diagnosing why AI search isn't citing your content, or auditing crawl patterns to find optimization gaps. |
| license | MIT |
| metadata | {"author":"ClawFu","version":"1.0.0","mcp-server":"@clawfu/mcp-skills"} |
Analyze server logs to understand how AI crawlers retrieve your content, then optimize placement and structure for maximum citation probability. Based on Metehan Yeşilyurt's log file analysis framework.
Use this skill when you need to:
This skill is particularly valuable for:
Source Expert: Metehan Yeşilyurt (SEO Consultant, speaker at BrightonSEO, The Search Session)
Core Thesis: Log file analysis reveals the real mechanics of AI search retrieval. Different AI products (AI Overviews, AI Mode, Perplexity) use fundamentally different retrieval pipelines, so optimizing for "AI search" requires understanding each product's specific crawl and retrieval behavior.
"AI Overviews, AI Mode, and Web Guide are three completely different products with different retrieval behaviors. You can't optimize for 'AI search' as if it's one thing." — Metehan Yeşilyurt, The Search Session
Key Insight: The shift from traditional SEO to AI search optimization is from chasing clicks in deterministic SERPs to chasing citations in machine-generated text. Log files reveal the mechanics behind citation selection.
| Claude Does | You Decide |
|---|---|
| Guides log analysis methodology | Which log data to provide |
| Identifies patterns in crawl behavior | Strategic priority of AI products |
| Recommends content placement optimizations | Content creation/modification scope |
| Maps retrieval behavior to optimization actions | Resource allocation for changes |
| Creates audit templates and checklists | Which recommendations to implement |
When invoked, I will guide you through:
| Bot User Agent | Operator | Purpose | Respects robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Training data + ChatGPT Browse | Yes |
| ChatGPT-User | OpenAI | Real-time browsing in ChatGPT | Yes |
| OAI-SearchBot | OpenAI | SearchGPT / ChatGPT Search | Yes |
| ClaudeBot | Anthropic | Training data collection | Yes |
| PerplexityBot | Perplexity | Real-time search + answer generation | Yes (mostly) |
| Google-Extended | Gemini/AI training (deprecated — now part of Googlebot) | N/A | |
| Googlebot | Traditional crawl + AI Overviews + AI Mode | Yes | |
| Bytespider | ByteDance | Training for TikTok AI features | Yes |
| CCBot | Common Crawl | Open dataset used by many AI models | Yes |
| Applebot-Extended | Apple | Apple Intelligence features | Yes |
# Extract all AI bot hits from Apache/Nginx access logs
grep -iE "(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|PerplexityBot|Bytespider|CCBot|Applebot-Extended)" access.log > ai_bots.log
# Count hits by bot
awk -F'"' '{print $6}' ai_bots.log | grep -oE "(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|PerplexityBot|Bytespider|CCBot)" | sort | uniq -c | sort -rn
# Top pages crawled by AI bots
awk '{print $7}' ai_bots.log | sort | uniq -c | sort -rn | head -50
# Crawl frequency by day
awk '{print $4}' ai_bots.log | cut -d: -f1 | tr -d '[' | sort | uniq -c
# Response codes for AI bots (are they getting 200s or errors?)
awk '{print $9}' ai_bots.log | sort | uniq -c | sort -rn
| Pattern | What It Means | Action |
|---|---|---|
| Bot fetches page frequently | Page is in retrieval index, content matters | Optimize this page first |
| Bot fetches page once then stops | Page was evaluated and deprioritized | Improve content quality/freshness |
| Bot never fetches a page | Page not discovered or blocked | Check internal linking, sitemap, robots.txt |
| Bot gets 404/500 | Technical issue blocking retrieval | Fix immediately |
| Bot fetches but doesn't cite | Content retrieved but not selected for answers | Improve structure, uniqueness, authority |
AI bots have crawl budgets like traditional bots. If your site is large:
QUESTIONS TO ANSWER:
1. What % of pages are being crawled by AI bots?
2. Are high-value pages being fetched?
3. Are AI bots wasting time on low-value pages (tag pages, pagination)?
4. What's the crawl frequency — daily? weekly? sporadic?
5. Do different AI bots prioritize different pages?
Map how each bot interacts with your site differently:
ANALYSIS TEMPLATE:
GPTBot:
Pages fetched: ___
Top pages: ___
Crawl frequency: ___
Response codes: ___
PerplexityBot:
Pages fetched: ___
Top pages: ___
Crawl frequency: ___
Response codes: ___
ClaudeBot:
Pages fetched: ___
Top pages: ___
Crawl frequency: ___
Response codes: ___
Each AI product uses a different retrieval pipeline. Optimizing requires understanding the mechanics.
Perplexity uses high embedding similarity to match content to queries, then actively diversifies sources.
What this means for you:
AI Overviews sit on top of existing Google Search rankings. Your traditional SEO signals matter.
What this means for you:
AI Mode decomposes complex questions into sub-queries, retrieves for each, then synthesizes.
What this means for you:
Understanding known LLM processing weaknesses helps you position content for maximum retrieval.
| Vulnerability | Description | Optimization |
|---|---|---|
| Recency bias | LLMs weight recent information higher, even when older info is more accurate | Date your content clearly. Update regularly. Recent timestamps = retrieval advantage |
| Lost-in-the-middle | LLMs process beginnings and endings of context better than middles | Place key claims, definitions, and statistics at the START and END of pages/sections |
| Data poisoning susceptibility | LLMs can be influenced by coordinated content patterns | Not actionable for you, but be aware: competitors can game AI answers |
| Prompt injection | LLMs can be manipulated via hidden instructions | Not actionable for SEO, but relevant for security audits |
This is the most actionable vulnerability for SEO:
PAGE STRUCTURE FOR LLM RETRIEVAL:
TOP (HIGH ATTENTION):
├── Key definition / direct answer
├── Critical statistics with context
└── Main claim / thesis
MIDDLE (LOW ATTENTION):
├── Supporting details
├── Tables and structured data (LLMs handle these well regardless of position)
├── Examples and elaboration
└── Supporting evidence
BOTTOM (HIGH ATTENTION):
├── Summary of key points
├── FAQ with direct answers
├── Conclusion with standalone claims
└── Updated date
"FAQs at the end of the page, tables in the middle — that's the lost-in-the-middle fix. LLMs handle structured data well regardless of position, but flowing text in the middle gets deprioritized." — Metehan Yeşilyurt
# AI Bot Log Audit: [Site Name]
**Date:** [Date]
**Period analyzed:** [Date range]
**Log source:** [Apache/Nginx/CDN]
## Executive Summary
[2-3 sentence overview of findings]
## Bot Activity Overview
| Bot | Total Hits | Unique Pages | Avg Daily Hits | Status |
|-----|-----------|-------------|----------------|--------|
| GPTBot | ___ | ___ | ___ | Active/Inactive |
| PerplexityBot | ___ | ___ | ___ | Active/Inactive |
| ClaudeBot | ___ | ___ | ___ | Active/Inactive |
| ChatGPT-User | ___ | ___ | ___ | Active/Inactive |
## Top Pages by AI Bot Activity
1. [URL] — [hits] hits — [which bots]
2. [URL] — [hits] hits — [which bots]
3. [URL] — [hits] hits — [which bots]
## Pages Not Crawled (Expected High-Value)
- [URL] — Likely cause: [blocked/not linked/low authority]
- [URL] — Likely cause: [___]
## Technical Issues
- [N] pages returning 404 to AI bots
- [N] pages returning 500 to AI bots
- robots.txt blocking: [yes/no — which bots?]
## Content Optimization Recommendations
### Priority 1: Fix Technical Issues
- [ ] [Specific fix]
### Priority 2: Optimize High-Traffic AI Pages
- [ ] Apply lost-in-the-middle structure to [page]
- [ ] Add entity schema to [page]
- [ ] Update freshness signals on [page]
### Priority 3: Improve Discoverability
- [ ] Internal link to [uncrawled page] from [crawled hub]
- [ ] Add to sitemap: [page]
- [ ] Create content for [gap topic]
## robots.txt Recommendations
[Current AI bot directives and recommended changes]
Context: News publisher noticed declining traffic from AI referrals. Log audit reveals:
Findings:
Actions:
Context: SaaS company publishes weekly blog but never appears in ChatGPT or Perplexity answers.
Findings:
Actions:
ACCESS CHECK
[ ] AI bots not blocked in robots.txt (check each bot separately)
[ ] AI bots getting 200 responses (not 403/404/500)
[ ] Sitemap accessible and up to date
[ ] Key pages present in sitemap
CRAWL PATTERNS
[ ] High-value pages are being crawled by AI bots
[ ] Crawl frequency is consistent (not declining)
[ ] AI bots not wasting budget on low-value pages
[ ] Multiple AI bots active (not just one)
CONTENT OPTIMIZATION
[ ] Key claims at beginning and end of pages (not buried in middle)
[ ] Tables and structured data for comparisons
[ ] FAQ sections at bottom of major pages
[ ] Content is updated with recent dates
[ ] Unique data or original insights present
ENTITY SIGNALS
[ ] Author/organization schema on relevant pages
[ ] Entity mentioned consistently across the site
[ ] Cross-platform entity presence established
Primary Sources:
Tools Referenced:
Technical References:
name: ai-bot-log-audit
category: seo-tools
subcategory: geo
version: 1.0.0
author: GUIA
source_expert: Metehan Yeşilyurt (SEO Consultant, BrightonSEO speaker) — The Search Session podcast
difficulty: intermediate
mode: centaur
tags: [ai-bots, log-analysis, geo, ai-search, crawl-audit, perplexity, gptbot, claudebot, retrieval, lost-in-the-middle]
created: 2026-02-10
updated: 2026-02-10
Run a structured 5-pass finishing audit on any website before launch — scoring visual polish, technical foundation, UX completeness, content quality, and cross-device readiness on 100 points. Use when: **Pre-launch** - Final validation before going live; **Post-redesign** - Verify nothing broke during the overhaul; **Client handoff** - Structured proof that the site is ready; **Quarterly review** - Catch accumulated debt; **Single-pass focus** - Run just Pass 2 after a perf sprint
Use when rewriting AI-generated text to match a specific person's or brand's authentic voice. Use when AI output sounds generic, corporate, or detectable. Unlike generic "humanizers," this skill requires voice analysis input and produces voice-consistent output, not fake imperfections.
Process large codebases (>100 files) using the Recursive Language Model pattern. Orchestrates parallel sub-agents to map-reduce across files without context rot. Use when: analyzing large repositories; auditing security or auth across many files; finding patterns across 50+ files; processing large log files or data dumps
Model best-case, worst-case, and likely revenue scenarios with sensitivity analysis for strategic planning. Use when: building financial forecasts; presenting board scenarios; planning headcount around revenue uncertainty; modeling pricing changes impact; preparing investor updates with upside/downside ranges
Use when optimizing for AI search visibility where individual experts and entities outperform generic brand pages. Use when building cross-platform authority for people, organizations, or products that AI models should recognize and cite.
Analyze paid advertising performance across channels and recommend budget reallocation to maximize ROAS and minimize CAC. Use when: planning quarterly ad budget allocation, diagnosing underperforming ad channels, deciding whether to scale spend on a channel, calculating marginal ROI across Google Ads, Meta, LinkedIn, or TikTok, rebalancing media mix after performance shifts, or setting up a test-and-scale framework for new channels.