ai-visibility-monitor

name

ai-visibility-monitor

description

Monitors brand visibility in AI responses (ChatGPT, Gemini, Perplexity) and community platforms (Reddit, Quora) across the customer funnel. Generates targeted monitoring questions and organizes collected data in session-based folders. Use when tracking AI recommendations for brands, collecting community discussions, testing if AI mentions a brand for use cases, measuring brand awareness across platforms, or analyzing competitive visibility. Each monitoring session creates an organized folder structure for questions, AI responses, and community data.

AI Visibility Monitor

Monitor brand visibility in AI platforms and community discussions across the customer journey.

What This Skill Does

1. Collects Community Data - Crawls Reddit and Quora for real user discussions
2. Analyzes Community Patterns - Extracts themes, comparisons, and user language
3. Generates AI Monitoring Questions - Creates questions derived from actual community discussions
4. Organizes Session Data - Structured folder with all collected data and analysis

Note: This skill generates questions for external AI crawlers. The AI crawling step is separate.

CRITICAL: Correct Workflow Order

❌ WRONG: Generate questions → Crawl community
✅ CORRECT: Crawl community → Analyze patterns → Generate questions

Why this order matters:

• Generic template questions miss real user language and concerns
• Community data reveals actual comparisons users make (e.g., "Arzopa vs Uperfect" not "Arzopa vs ASUS")
• Real pain points surface from community (e.g., "MacBook M3 compatibility issues")
• User personas become clear (e.g., Steam Deck gamers, digital nomads)

Session Folder Structure

monitoring_sessions/
└── {brand}_{YYYYMMDD_HHMMSS}/
    ├── questions/
    │   └── monitoring_questions.yaml      # AI monitoring questions (generated LAST)
    ├── ai_responses/
    │   ├── chatgpt/                       # AI crawler output (external)
    │   ├── gemini/
    │   └── perplexity/
    ├── community_data/
    │   ├── reddit/
    │   │   └── comprehensive_crawl.json   # All Reddit posts
    │   └── quora/
    │       └── comprehensive_crawl.json   # All Quora questions
    ├── analysis/
    │   ├── brand_research.md              # Products, competitors, audiences
    │   ├── community_analysis.md          # Themes, patterns from community
    │   └── summary.md                     # Executive summary
    └── session_metadata.json

Workflow

Step 1: Initialize Session

Create session folder and metadata:

{
  "session_id": "{brand}_{timestamp}",
  "brand": "Brand Name",
  "website": "brand.com",
  "created_at": "2026-01-29T10:30:00Z",
  "status": "initialized",
  "platforms": {
    "ai": ["chatgpt", "gemini", "perplexity"],
    "community": ["reddit", "quora"]
  }
}

Step 2: Brand Research

Research the target brand to understand:

• Product portfolio (models, prices, features)
• Target audiences (who buys this?)
• Key differentiators
• Main competitors

Output: analysis/brand_research.md

Step 3: Crawl Reddit (BEFORE generating questions)

Use the Playwright-based Reddit crawler to collect discussions:

from crawlers.reddit import RedditCrawler

crawler = RedditCrawler()

# Search multiple subreddits and keywords
result = await crawler.comprehensive_search(
    keywords=["Arzopa", "portable monitor", "USB-C display", "travel monitor"],
    subreddits=["digitalnomad", "SteamDeck", "monitors", "remotework", "battlestations"],
    time_filter="year",
    limit_per_subreddit=100
)

Output: community_data/reddit/comprehensive_crawl.json

Important: The Reddit crawler uses Playwright browser automation, not PRAW API (Reddit blocked API access).

Step 4: Crawl Quora (BEFORE generating questions)

Use the Playwright-based Quora crawler:

from crawlers.quora import QuoraCrawler

crawler = QuoraCrawler()

# Search via Google site search
result = await crawler.search_questions(
    keywords=["best portable monitor", "portable monitor recommendation", "USB-C portable display"],
    limit_per_keyword=30
)

Output: community_data/quora/comprehensive_crawl.json

Step 5: Analyze Community Data

Review collected data and extract:

1. Use case categories - How do users talk about this product?
2. Technical questions - What concerns do users have?
3. Brand mentions - Direct mentions of target brand
4. Competitor mentions - Which competitors appear in discussions?
5. Decision factors - What do users care about when buying?
6. Pain points - What problems do users face?
7. Question patterns - How do users phrase their questions?

Output: analysis/community_analysis.md

Step 6: Generate Monitoring Questions (LAST)

Now generate questions based on community insights:

1. Read references/funnel-templates.yaml for structure
2. Replace generic placeholders with real community language
3. Use actual comparisons found in community (not assumed ones)
4. Include technical/compatibility questions from real pain points

Question Sources:

• ~70% derived from community data (real user questions)
• ~30% from funnel templates (awareness, competitor stages)

Output: questions/monitoring_questions.yaml

Step 7: Update Session Metadata

Update session_metadata.json with:

• Status: ready_for_ai_crawling
• Question counts by funnel stage
• Community data counts
• Workflow completion flags

Step 8: Create Summary

Generate analysis/summary.md with:

• Executive summary
• Key insights from community
• Question generation methodology
• Next steps for AI crawling

Reddit Crawler (Playwright-based)

The Reddit crawler uses Playwright browser automation to bypass API restrictions.

Usage

from crawlers.reddit import RedditCrawler

async with RedditCrawler() as crawler:
    # Search a subreddit
    posts = await crawler.search_subreddit(
        subreddit="digitalnomad",
        query="portable monitor",
        sort="relevance",  # relevance, hot, new, top
        time_filter="year",
        limit=50
    )
    
    # Comprehensive search across multiple subreddits
    all_posts = await crawler.comprehensive_search(
        keywords=["Arzopa", "portable monitor"],
        subreddits=["digitalnomad", "SteamDeck", "monitors"],
        time_filter="year",
        limit_per_subreddit=100
    )

Output Format

{
  "platform": "reddit",
  "collected_at": "2026-01-29T21:00:00Z",
  "search_config": {
    "keywords": ["Arzopa", "portable monitor"],
    "subreddits": ["digitalnomad", "SteamDeck"],
    "time_filter": "year"
  },
  "posts": [
    {
      "post_id": "abc123",
      "subreddit": "digitalnomad",
      "title": "Best portable monitor for travel?",
      "url": "https://reddit.com/r/digitalnomad/comments/abc123",
      "score": 45,
       
       
       
    
  
   
     
     
  

Rate Limiting

• Add 2-3 second delays between page loads
• Reddit may serve CAPTCHAs for aggressive scraping
• Run during off-peak hours for better reliability

Quora Crawler (Playwright-based)

Uses Google site search to find Quora questions.

Usage

from crawlers.quora import QuoraCrawler

async with QuoraCrawler() as crawler:
    questions = await crawler.search_questions(
        keywords=["best portable monitor", "Arzopa monitor review"],
        limit_per_keyword=30
    )

Output Format

{
  "platform": "quora",
  "collected_at": "2026-01-29T21:30:00Z",
  "search_keywords": ["best portable monitor", "Arzopa"],
  "questions": [
    {
      "title": "What is the best portable monitor for remote work?",
      "url": "https://www.quora.com/What-is-the-best-portable-monitor-for-remote-work",
      "source": "google_site_search"
    }
  ],
  "summary": {
    "total_questions": 89
  }
}

Reference Files

• funnel-templates.yaml: Question templates organized by funnel stage (awareness, interest, purchase, competitor)
• community-sources.md: Guide for identifying relevant communities

Tips

1. Crawl community FIRST - This is the most important step
2. Analyze before generating - Extract real patterns, don't assume
3. Use real user language - "single USB-C cable" not "USB-C connectivity"
4. Include actual comparisons - From community, not marketing assumptions
5. Generate 50+ questions - Cover all funnel stages comprehensively
6. Include technical questions - From real pain points found in community

Environment Setup

# Python 3.10+ required
pip install playwright
playwright install chromium

# Browser path (optional - uses Playwright default if not set)
export BROWSER_EXECUTABLE_PATH=""

# For headless operation
export BROWSER_HEADLESS=true

Standalone Scripts

The skill includes standalone Python scripts for session initialization and community data collection. These scripts work independently without requiring the gptAutoCrawling project.

init_session.py

Initialize a new monitoring session with proper folder structure:

python init_session.py \
    --brand "BrandName" \
    --output-dir ./monitoring_sessions \
    --website "brand.com" \
    --category "product category"

Flags:

• --brand (required): Brand name to monitor
• --output-dir (default: ./monitoring_sessions): Base directory for sessions
• --website (optional): Brand website URL
• --category (optional): Product category

reddit_crawl.py

Crawl Reddit for brand discussions using Playwright browser automation:

python reddit_crawl.py \
    --brand "Arzopa" \
    --keywords "Arzopa,portable monitor,travel display" \
    --subreddits "digitalnomad,SteamDeck,monitors,battlestations" \
    --output ./monitoring_sessions/arzopa_20260129/community_data/reddit/crawl.json \
    --time-filter year \
    --limit 30

Flags:

• --brand (required): Brand name for metadata
• --keywords (required): Comma-separated search keywords
• --subreddits (optional): Comma-separated subreddits (global search if omitted)
• --output (required): Output JSON file path
• --time-filter (default: month): hour, day, week, month, year, all
• --limit (default: 30): Max posts per keyword/subreddit combo
• --delay (default: 2): Seconds between requests
• --headless (default: true): Run browser headless

quora_crawl.py

Crawl Quora questions via Google site search:

python quora_crawl.py \
    --brand "Arzopa" \
    --keywords "best portable monitor,portable monitor recommendation" \
    --output ./monitoring_sessions/arzopa_20260129/community_data/quora/crawl.json \
    --limit 20

Flags:

• --brand (required): Brand name for metadata
• --keywords (required): Comma-separated search keywords
• --output (required): Output JSON file path
• --limit (default: 20): Max questions per keyword
• --delay (default: 3): Seconds between requests
• --headless (default: true): Run browser headless

Note: Quora crawl uses Google site search, which may be blocked. If blocked, the script will return an empty questions array with errors logged.

Typical CLI Workflow

# 1. Initialize session
python scripts/init_session.py --brand "Arzopa" --output-dir ./monitoring_sessions

# Find the created session folder
SESSION_DIR=$(ls -d ./monitoring_sessions/arzopa_* | tail -1)

# 2. Crawl Reddit
python scripts/reddit_crawl.py \
    --brand "Arzopa" \
    --keywords "Arzopa,portable monitor" \
    --subreddits "digitalnomad,SteamDeck,monitors" \
    --output "$SESSION_DIR/community_data/reddit/crawl.json" \
    --time-filter year

# 3. Crawl Quora
python scripts/quora_crawl.py \
    --brand "Arzopa" \
    --keywords "best portable monitor" \
    --output "$SESSION_DIR/community_data/quora/crawl.json"

# 4. Review collected data
cat "$SESSION_DIR/community_data/reddit/crawl.json" | jq '.summary'
cat "$SESSION_DIR/community_data/quora/crawl.json" | jq '.summary'

Troubleshooting

Reddit: "Timeout" or "Page not loading"

• Check network connectivity
• Increase page timeout
• Try running in non-headless mode to debug

Reddit: "CAPTCHA detected"

• Add longer delays between requests
• Try again later
• Use different IP/proxy

Quora: "No results found"

• Try different keyword variations
• Google may be blocking automated requests
• Add longer delays between searches

Browser crashes

• Check available memory
• Reduce concurrent operations
• Clear Playwright cache: playwright install --clean