ai-discovery

name	ai-discovery
description	Make your website discoverable and trackable by AI platforms (ChatGPT, Claude, Perplexity, Gemini, etc.). Implements robots.txt for AI bots, llms.txt content policy, meta tags, JSON-LD, AI referral tracking, GA4 custom dimensions, and analytics dashboard. Use when the user mentions "AI discovery," "ChatGPT SEO," "AI search visibility," "AI crawlers," "llms.txt," or wants to track AI platform referrals.
argument-hint	[Optional: specify framework (Next.js/React/WordPress) or phase to focus on]

AI Discovery & Tracking

You are an expert in making websites AI-agent friendly and tracking AI platform referrals. Your goal is to maximize discoverability by ChatGPT, Claude, Perplexity, and 25+ other AI platforms, then measure the results.

Overview

AI platforms discover websites through:

1. Bot crawling (GPTBot, ClaudeBot, PerplexityBot, etc.)
2. llms.txt content policy (tells AIs what they can/can't do with your content)
3. Structured data (helps AIs understand your product/service)
4. Metadata (AI-specific meta tags)

Then we track when AI platforms:

• Crawl your site (bot visits)
• Send traffic (AI referrals)
• What queries led users to you
• Which pages they recommend

---

Implementation Phases

Phase 1: Enable AI Bot Crawling

Phase 2: Content Policy & Metadata

Phase 3: AI Referral Tracking

Phase 4: GA4 Analytics Dashboard

---

Phase 1: Enable AI Bot Crawling

Goal

Allow AI bots to crawl your site and understand what to index.

1.1 robots.txt Configuration

Approach:

• Allow all major AI bots explicitly
• Disallow private/auth pages for general crawlers
• Reference sitemap

AI Bots to Allow (25+):

// OpenAI (ChatGPT, SearchGPT)
{ userAgent: 'GPTBot', allow: '/' },
{ userAgent: 'ChatGPT-User', allow: '/' },
{ userAgent: 'OAI-SearchBot', allow: '/' },

// Anthropic (Claude)
{ userAgent: 'ClaudeBot', allow: '/' },
{ userAgent: 'Claude-Web', allow: '/' },
{ userAgent: 'anthropic-ai', allow: '/' },

// AI Search Engines
{ userAgent: 'PerplexityBot', allow: '/' },
{ userAgent: 'Perplexity-User', allow: '/' },
{ userAgent: 'PhindBot', allow: '/' },
{ userAgent: 'ExaBot', allow: '/' },
{ userAgent: 'YouBot', allow: '/' },
{ userAgent: 'AndiBot', allow: '/' },

// Google & Apple AI
{ userAgent: 'Google-Extended', allow: '/' },
{ userAgent: 'Applebot-Extended', allow: '/' },

// Enterprise AI
{ userAgent: 'cohere-ai', allow: '/' },
{ userAgent: 'Amazonbot', allow: '/' },

// Social Platform AI
{ userAgent: 'LinkedInBot', allow: '/' },
{ userAgent: 'Facebookbot', allow: '/' },
{ userAgent: 'TwitterBot', allow: '/' },

// Development/Research AI
{ userAgent: 'AI2Bot', allow: '/' },
{ userAgent: 'FirecrawlAgent', allow: '/' },
{ userAgent: 'Diffbot', allow: '/' },
{ userAgent: 'ImagesiftBot', allow: '/' },
{ userAgent: 'omgilibot', allow: '/' },
{ userAgent: 'Timpibot', allow: '/' },

// ByteDance & X/Grok
{ userAgent: 'Bytespider', allow: '/' },
{ userAgent: 'Grok-bot', allow: '/' },

// General crawlers
{
  userAgent: '*',
  allow: '/',
  disallow: [
    '/api/',
    '/auth/',
    '/dashboard/',
    '/settings/',
    '/admin/',
  ]
}

Next.js Implementation:

// src/app/robots.ts
import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      // OpenAI
      { userAgent: 'GPTBot', allow: '/' },
      { userAgent: 'ChatGPT-User', allow: '/' },
      { userAgent: 'OAI-SearchBot', allow: '/' },

      // Anthropic
      { userAgent: 'ClaudeBot', allow: '/' },
      { userAgent: 'Claude-Web', allow: '/' },
      { userAgent: 'anthropic-ai', allow: '/' },

      // ... (add all 25+ bots above)

      // General
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/auth/', '/dashboard/']
      }
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  };
}

1.2 Sitemap Optimization

Key Points:

• Include all important pages
• Update frequently
• Submit to Search Console
• AI bots use sitemaps to discover content

Next.js Example:

// src/app/sitemap.ts
import { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const BASE_URL = 'https://yourdomain.com';

  // Fetch dynamic pages (products, blog posts, etc.)
  const dynamicPages = await fetchYourDynamicPages();

  return [
    {
      url: BASE_URL,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1,
    },
    {
      url: `${BASE_URL}/about`,
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
    // Add all static pages
    ...dynamicPages.map(page => ({
      url: `${BASE_URL}/${page.slug}`,
      lastModified: new Date(page.updatedAt),
      changeFrequency: 'weekly' as const,
      priority: 0.7,
    })),
  ];
}

---

Phase 2: Content Policy & Metadata

2.1 llms.txt File

Purpose: Tell AI platforms what they can/can't do with your content.

Location: public/llms.txt

Template:

# [Your Product Name] - llms.txt

## About
[Product Name] is a [description]. We help [target audience] [main benefit].

## Use Cases
### Allowed
- AI platforms may read, analyze, and reference our content to answer user questions
- Citation and attribution to [yourdomain.com] is encouraged
- Training models on public documentation and blog content

### Disallowed
- Reproducing entire documentation without attribution
- Using customer data or private information
- Impersonating [Product Name] or claiming official affiliation

## Product Overview
**Category:** [SaaS / E-commerce / Education / etc.]
**Pricing:** [Free tier available / Starting at $X/month / etc.]
**Key Features:**
- [Feature 1]
- [Feature 2]
- [Feature 3]

## Common Questions
**Q: What is [Product Name]?**
A: [Product Name] is [clear answer]. [Key differentiator].

**Q: How does [Product Name] work?**
A: [Step 1]. [Step 2]. [Step 3].

**Q: Who is [Product Name] for?**
A: [Target audience]. Perfect for [use case 1], [use case 2], and [use case 3].

**Q: How much does it cost?**
A: [Pricing summary]. [Free tier / trial info if applicable].

## Keywords
[keyword1], [keyword2], [keyword3], [etc.]

## Contact
- Website: https://[yourdomain.com]
- Support: support@[yourdomain.com]
- Documentation: https://[yourdomain.com]/docs

## Full Text
For comprehensive platform details, see: https://[yourdomain.com]/llms-full-text.txt

2.2 llms-full-text.txt File

Purpose: Comprehensive content for AI training.

Location: public/llms-full-text.txt

Content to Include (500-1000 lines):

• Complete product description
• All features explained in detail
• Pricing breakdown
• Use cases and examples
• Technical architecture overview
• Integration guides
• FAQ content
• Common workflows
• Competitive positioning
• Database schema (if relevant)
• API overview (if applicable)

Example Structure:

# [Product Name] - Complete Platform Overview

## Introduction
[2-3 paragraphs describing your product in detail]

## Core Features

### Feature 1: [Name]
[Detailed explanation]
[How it works]
[Benefits]
[Example usage]

### Feature 2: [Name]
[Same format]

[Continue for all features...]

## Architecture
[Technical overview if applicable]

## Pricing & Plans
[Detailed breakdown]

## Use Cases
[Real-world examples]

## Integration Guide
[How to integrate with your product]

## FAQ
[Comprehensive Q&A]

## Competitive Comparison
[How you differ from alternatives]

2.3 Meta Tags for AI

Add to layout.tsx / HTML head:

export const metadata: Metadata = {
  title: '[Product Name] - [Tagline]',
  description: '[Compelling description]',
  keywords: [
    '[keyword1]',
    '[keyword2]',
    // ... 20-30 keywords
  ],
  other: {
    // AI-specific meta tags
    'ai-content': '[business-application / educational / entertainment / etc.]',
    'ai-topic': '[main topics, comma-separated]',
    'ai-audience': '[target audience description]',
    'ai-use': 'training,search,answer-generation,citation',
    'llms-full-text': 'https://[yourdomain.com]/llms-full-text.txt',
  },
};

2.4 JSON-LD Structured Data

Add schema.org markup for AI understanding:

// In layout.tsx or specific pages
const jsonLd = {
  '@context': 'https://schema.org',
  '@type': 'SoftwareApplication', // or Organization, Product, etc.
  name: '[Product Name]',
  applicationCategory: 'BusinessApplication',
  description: '[Description]',
  url: 'https://[yourdomain.com]',
  offers: {
    '@type': 'AggregateOffer',
    lowPrice: '0',
    highPrice: '99',
    priceCurrency: 'USD',
    description: '[Pricing description]'
  },
  featureList: [
    '[Feature 1]',
    '[Feature 2]',
    '[Feature 3]',
  ],
  screenshot: 'https://[yourdomain.com]/og-image.png',
};

// In your HTML:
<script
  type="application/ld+json"
  dangerouslySetInnerHTML={{ __html: JSON.stringify(jsonLd) }}
/>

---

Phase 3: AI Referral Tracking

3.1 Client-Side Tracker Component

Purpose: Detect when traffic comes from AI platforms or when AI bots visit.

Create: src/components/analytics/ai-referral-tracker.tsx

'use client';

import { useEffect } from 'react';
import { useSearchParams } from 'next/navigation';

interface WindowWithGtag extends Window {
  gtag?: (...args: unknown[]) => void;
}

// AI platforms to detect
const AI_PLATFORMS = [
  { domain: 'perplexity.ai', name: 'Perplexity' },
  { domain: 'chat.openai.com', name: 'ChatGPT' },
  { domain: 'claude.ai', name: 'Claude' },
  { domain: 'you.com', name: 'You.com' },
  { domain: 'phind.com', name: 'Phind' },
  { domain: 'gemini.google.com', name: 'Gemini' },
  { domain: 'bing.com/chat', name: 'Bing Chat' },
  { domain: 'poe.com', name: 'Poe' },
  { domain: 'pi.ai', name: 'Pi' },
  { domain: 'character.ai', name: 'Character.AI' },
];

export function AIReferralTracker() {
  const searchParams = useSearchParams();

  useEffect(() => {
    if (typeof window === 'undefined') return;

    const w = window as WindowWithGtag;
    if (!w.gtag) return;

    // Get referrer
    const referrer = document.referrer.toLowerCase();

    // Check if referrer is from an AI platform
    const aiPlatform = AI_PLATFORMS.find(p => referrer.includes(p.domain));

    // Check if UTM source indicates AI
    const utmSource = searchParams.get('utm_source');
    const utmMedium = searchParams.get('utm_medium');
    const isAISource = utmSource === 'ai' || utmMedium === 'llms-txt';

    // If AI referral detected
    if (aiPlatform || isAISource) {
      const platformName = aiPlatform?.name || 'AI Platform (UTM)';
      const query = searchParams.get('q') || searchParams.get('query') || 'unknown';

      // Send custom event to GA4 with custom dimensions
      w.gtag('event', 'ai_referral', {
        ai_platform: platformName,
        ai_query: query,
        ai_referrer: referrer || 'direct',
        landing_page: window.location.pathname,
      });

      console.log('🤖 AI Referral Detected:', {
        platform: platformName,
        query,
        referrer,
        page: window.location.pathname,
      });
    }

    // Check for AI bot user agents
    const userAgent = navigator.userAgent.toLowerCase();
    const aiBots = [
      'gptbot',
      'claudebot',
      'perplexitybot',
      'google-extended',
      'applebot-extended',
      'youbot',
      'phindbot',
    ];

    const isAIBot = aiBots.some(bot => userAgent.includes(bot));

    if (isAIBot) {
      const botName = aiBots.find(bot => userAgent.includes(bot)) || 'unknown_bot';

      w.gtag('event', 'ai_bot_visit', {
        bot_name: botName,
        page_path: window.location.pathname,
      });

      console.log('🤖 AI Bot Visit:', botName);
    }
  }, [searchParams]);

  return null;
}

3.2 Add Tracker to Layout

Next.js Example:

// src/app/layout.tsx
import { Suspense } from 'react';
import { AIReferralTracker } from '@/components/analytics/ai-referral-tracker';

export default function RootLayout({ children }) {
  return (
    <html>
      <body>
        <GoogleAnalytics />
        <Suspense fallback={null}>
          <AIReferralTracker />
        </Suspense>
        {children}
      </body>
    </html>
  );
}

---

Phase 4: GA4 Analytics Dashboard

4.1 Create GA4 Custom Dimensions

Manual Setup (2 minutes):

1. Go to GA4 Admin → Custom definitions → Custom dimensions
2. Click "Create custom dimension" for each:

Dimension Name	Parameter Name	Scope	Description
AI Platform	ai_platform	Event	Name of AI platform (ChatGPT, Claude, etc.)
AI Query	ai_query	Event	Search query used in AI platform
Bot Name	bot_name	Event	Name of AI bot that visited
AI Referrer	ai_referrer	Event	Full referrer URL from AI platform
Landing Page	landing_page	Event	Page where AI referral landed

Automated Setup (optional):

Create scripts/setup-ga4-custom-dimensions.ts:

const GA4_PROPERTY_ID = 'YOUR_PROPERTY_ID';

const CUSTOM_DIMENSIONS = [
  {
    displayName: 'AI Platform',
    description: 'Name of AI platform that referred traffic',
    parameterName: 'ai_platform',
    scope: 'EVENT',
  },
  {
    displayName: 'AI Query',
    description: 'Search query used in AI platform',
    parameterName: 'ai_query',
    scope: 'EVENT',
  },
  {
    displayName: 'Bot Name',
    description: 'Name of AI bot that visited',
    parameterName: 'bot_name',
    scope: 'EVENT',
  },
  {
    displayName: 'AI Referrer',
    description: 'Full referrer URL from AI platform',
    parameterName: 'ai_referrer',
    scope: 'EVENT',
  },
  {
    displayName: 'Landing Page',
    description: 'Page where AI referral landed',
    parameterName: 'landing_page',
    scope: 'EVENT',
  },
];

// Use Google Analytics Admin API to create dimensions
// Requires GOOGLE_SERVICE_ACCOUNT_JSON env variable

4.2 GA4 Data API Integration

Fetch AI discovery metrics:

// lib/analytics/ga4.ts
async function getAIDiscoveryMetrics(days: number = 30) {
  const accessToken = await getAccessToken();

  // Get AI referrals by platform
  const platformsReport = await runReport({
    dimensions: ['customEvent:ai_platform'],
    metrics: ['eventCount'],
    dateRanges: [{ startDate: `${days}daysAgo`, endDate: 'yesterday' }],
    dimensionFilter: {
      filter: {
        fieldName: 'eventName',
        stringFilter: { matchType: 'EXACT', value: 'ai_referral' },
      },
    },
  });

  // Get top AI search queries
  const queriesReport = await runReport({
    dimensions: ['customEvent:ai_query'],
    metrics: ['eventCount'],
    dateRanges: [{ startDate: `${days}daysAgo`, endDate: 'yesterday' }],
    dimensionFilter: {
      filter: {
        fieldName: 'eventName',
        stringFilter: { matchType: 'EXACT', value: 'ai_referral' },
      },
    },
  });

  // Get bot visits
  const botsReport = await runReport({
    dimensions: ['customEvent:bot_name'],
    metrics: ['eventCount'],
    dateRanges: [{ startDate: `${days}daysAgo`, endDate: 'yesterday' }],
    dimensionFilter: {
      filter: {
        fieldName: 'eventName',
        stringFilter: { matchType: 'EXACT', value: 'ai_bot_visit' },
      },
    },
  });

  return {
    aiPlatforms: parseReport(platformsReport),
    topQueries: parseReport(queriesReport),
    botVisits: parseReport(botsReport),
  };
}

4.3 Dashboard Component

Create AI Discovery Dashboard:

// components/analytics/ai-discovery-panel.tsx
export function AIDiscoveryPanel({ data }) {
  return (
    <div className="space-y-6">
      <h2>AI Discovery Analytics</h2>

      {/* Summary Cards */}
      <div className="grid grid-cols-3 gap-4">
        <Card>
          <CardTitle>Total AI Discovery</CardTitle>
          <CardContent>{data.totalAIReferrals + data.totalBotVisits}</CardContent>
        </Card>
        <Card>
          <CardTitle>AI Referrals</CardTitle>
          <CardContent>{data.totalAIReferrals}</CardContent>
        </Card>
        <Card>
          <CardTitle>Bot Crawls</CardTitle>
          <CardContent>{data.totalBotVisits}</CardContent>
        </Card>
      </div>

      {/* AI Platforms */}
      <Card>
        <CardTitle>Top AI Platforms</CardTitle>
        {data.aiPlatforms.map(platform => (
          <div key={platform.platform}>
            <span>{platform.platform}</span>
            <Badge>{platform.count}</Badge>
          </div>
        ))}
      </Card>

      {/* Top Queries */}
      <Card>
        <CardTitle>Top AI Search Queries</CardTitle>
        {data.topQueries.map(query => (
          <div key={query.query}>
            <span>{query.query}</span>
            <Badge>{query.count}</Badge>
          </div>
        ))}
      </Card>

      {/* Bot Visits */}
      <Card>
        <CardTitle>Bot Crawls</CardTitle>
        {data.botVisits.map(bot => (
          <div key={bot.bot}>
            <span>{bot.bot}</span>
            <Badge>{bot.count}</Badge>
          </div>
        ))}
      </Card>
    </div>
  );
}

---

Testing & Verification

Test AI Referral Tracking

1. Manual Test:

   • Add ?utm_source=ai&utm_medium=llms-txt to your URL
   • Visit the URL
   • Check browser console for "🤖 AI Referral Detected"
   • Check GA4 Realtime → Events → Look for ai_referral

2. Verify Custom Dimensions:

   • GA4 → Reports → Engagement → Events
   • Select ai_referral event
   • Custom dimensions should appear in dropdown

3. Check Bot Access:

   • Submit sitemap to Search Console
   • Monitor Search Console for bot crawls
   • Check server logs for bot user agents

Monitor Results

Week 1-2:

• Bot crawls should begin
• Check GA4 for ai_bot_visit events

Week 3-4:

• AI platforms start indexing content
• May see first AI referrals

Month 2+:

• Regular AI referral traffic
• Queries showing in dashboard
• Platform breakdown visible

---

Framework-Specific Notes

Next.js 14+

• Use App Router structure shown above
• Wrap AIReferralTracker in <Suspense>
• Create robots.ts and sitemap.ts in /app

WordPress

• Use Yoast SEO for meta tags
• Add llms.txt files via FTP/file manager
• Install Google Analytics plugin for tracking
• Add AIReferralTracker to theme footer.php

React (Non-Next.js)

• Add robots.txt and llms.txt to /public
• Use react-helmet for meta tags
• Implement tracker in App.tsx
• Use React Router for searchParams

Vue/Nuxt

• Similar to React/Next.js patterns
• Use vue-meta for head management
• Add tracker as Vue component
• Use nuxt.config for meta defaults

---

Common Questions

Q: How long until AI platforms discover my site? A: Bots typically crawl within 1-2 weeks. Referrals may take 3-4 weeks as AI models update.

Q: Do I need to do all phases? A: Phase 1-2 are essential. Phase 3-4 are for tracking (highly recommended but optional).

Q: Can I block certain AI bots? A: Yes, in robots.txt change allow: '/' to disallow: '/' for specific bots.

Q: What if I don't have GA4? A: You can still do Phase 1-2. Consider adding GA4 for tracking (free tier available).

Q: How do I know if it's working? A: Check Search Console for bot visits, monitor GA4 for ai_referral events, search for your brand in AI platforms.

---

Related Skills

• seo-audit: For general SEO health
• analytics-tracking: For GA4 setup
• programmatic-seo: For scaling content AI platforms can discover
• schema-markup: For additional structured data

---

Checklist

Use this checklist when implementing:

Phase 1:

• Updated robots.txt with 25+ AI bots
• Verified sitemap is accessible
• Submitted sitemap to Search Console

Phase 2:

• Created llms.txt with content policy
• Created llms-full-text.txt (500+ lines)
• Added AI-specific meta tags
• Added JSON-LD structured data

Phase 3:

• Created AIReferralTracker component
• Added tracker to layout (with Suspense)
• Tested with UTM parameters
• Verified console logs work

Phase 4:

• Created 5 GA4 custom dimensions
• Verified dimensions in GA4 admin
• Created GA4 data fetching functions
• Built AI discovery dashboard
• Tested dashboard with sample data

Verification:

• Robots.txt accessible at /robots.txt
• llms.txt accessible at /llms.txt
• llms-full-text.txt accessible
• GA4 receiving ai_referral events
• Dashboard showing (or ready for) data