llm-public-opinion-analytics-assistant

# LLM-Based Intelligent Public Opinion Analytics Assistant > Skill by [ara.so](https://ara.so) — Data Skills collection. ## Overview This project is an intelligent public opinion analysis assistant that combines real-time data from **26 trending lists across 15 mainstream platforms** with large language model (LLM) analysis capabilities. It provides conversational hot search queries, topic-specific searches, topic clustering analysis, and sentiment analysis through a web interface. The system supports keyboard shortcuts for crawler control, multi-platform data retrieval with direct navigation, and multi-channel hot topic push notifications (email, WeChat, Enterprise WeChat, Telegram). ## Key Features - **Multi-Platform Data Collection**: Crawls 26 trending lists from 15 platforms - **LLM-Powered Analysis**: Topic clustering, sentiment analysis, and trend detection - **Conversational Interface**: Natural language queries for data exploration - **Video Content Analysis**: Extracts information even from video-based news - **Multi-Channel Notifications**: Email, WeChat Work, Telegram bot push notifications - **Crawler Control**: Quick start/stop via keyboard shortcuts - **Database Storage**: MySQL-based data persistence ## Installation ### Prerequisites **Browser Driver Setup** (Required for news detail extraction): 1. **Check browser version**: - Open Edge/Chrome → Settings → About - Note your version (e.g., `115.0.5790.102`) 2. **Download matching driver**: - Chrome: https://chromedriver.chromium.org/ - Edge: https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/ 3. **Install driver**: ```bash # Linux/macOS sudo mv chromedriver /usr/local/bin/ sudo chmod +x /usr/local/bin/chromedriver # Windows: Add driver directory to PATH # e.g., C:\WebDriver\chromedriver.exe ``` 4. **Verify installation**: ```bash chromedriver --version ``` ### Environment Setup ```bash # Clone repository git clone https://github.com/hmmnxkl/LLM-Based-Intelligent-Public-Opinion-Analytics-Assistant.git cd LLM-Based-Intelligent-Public-Opinion-Analytics-Assistant # Create virtual environment python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate # Install dependencies pip install -r requirements.txt ``` ### Database Configuration 1. **Install MySQL** (8.0+ recommended) 2. **Create database and tables**: ```python # Reference init.py for schema import mysql.connector conn = mysql.connector.connect( host='localhost', user='your_user', password='your_password' ) cursor = conn.cursor() # Create database cursor.execute("CREATE DATABASE IF NOT EXISTS hotsearch_db CHARACTER SET utf8mb4") cursor.execute("USE hotsearch_db") # Create tables (see init.py for full schema) cursor.execute(""" CREATE TABLE IF NOT EXISTS hot_searches ( id INT AUTO_INCREMENT PRIMARY KEY, platform VARCHAR(50), title VARCHAR(500), url VARCHAR(1000), rank INT, heat_value VARCHAR(100), timestamp DATETIME, content TEXT, sentiment VARCHAR(50), INDEX idx_platform_timestamp (platform, timestamp) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 """) conn.commit() cursor.close() conn.close() ``` ### Configuration Files **Create `.env` file in project root**: ```bash # Database Configuration MYSQL_HOST=localhost MYSQL_PORT=3306 MYSQL_USER=your_user MYSQL_PASSWORD=your_password MYSQL_DATABASE=hotsearch_db # LLM Configuration (OpenAI-compatible API) OPENAI_API_KEY=your_api_key_here OPENAI_BASE_URL=https://api.openai.com/v1 OPENAI_MODEL=gpt-4 # Or use Huawei Pangu Model (recommended for Chinese) # PANGU_API_KEY=your_pangu_key # PANGU_BASE_URL=your_pangu_endpoint # Push Notification Channels # Email (SMTP) SMTP_HOST=smtp.gmail.com SMTP_PORT=587 SMTP_USER=your_email@gmail.com SMTP_PASSWORD=your_app_password EMAIL_RECIPIENTS=recipient1@example.com,recipient2@example.com # Enterprise WeChat Bot WECHAT_WORK_WEBHOOK=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=your_key # Telegram Bot TELEGRAM_BOT_TOKEN=your_bot_token TELEGRAM_CHAT_ID=your_chat_id ``` **Crawler Configuration** (`hotsearchcrawler/settings.py`): ```python # MySQL settings MYSQL_HOST = 'localhost' MYSQL_PORT = 3306 MYSQL_USER = 'your_user' MYSQL_PASSWORD = 'your_password' MYSQL_DATABASE = 'hotsearch_db' # Optional: Platform-specific cookies COOKIES = { 'weibo': 'your_weibo_cookie', 'douyin': 'your_douyin_cookie' } # Concurrent requests CONCURRENT_REQUESTS = 16 DOWNLOAD_DELAY = 1 ``` ## Usage ### Starting the System **1. Launch the web application**: ```bash python app.py ``` Access at: `http://localhost:5000` **2. Start crawlers** (via web interface or CLI): ```bash # Manual crawler start for testing python run_spiders.py # Or test individual spider cd hotsearchcrawler scrapy crawl weibo_spider scrapy crawl bilibili_spider ``` ### Core API Usage #### Conversational Query Interface ```python from hotsearch_analysis_agent.agent import OpinionAnalysisAgent # Initialize agent agent = OpinionAnalysisAgent( api_key=os.getenv('OPENAI_API_KEY'), base_url=os.getenv('OPENAI_BASE_URL'), model=os.getenv('OPENAI_MODEL', 'gpt-4') ) # Query hot searches response = agent.query("Show me top trending topics about AI") print(response['analysis']) # Topic clustering clusters = agent.cluster_topics("人工智能", days=7) for cluster in clusters: print(f"Cluster: {cluster['theme']}") print(f"Articles: {len(cluster['articles'])}") print(f"Sentiment: {cluster['sentiment']}") # Sentiment analysis sentiment = agent.analyze_sentiment("特定主题关键词", platform="weibo") print(f"Positive: {sentiment['positive']}%") print(f"Negative: {sentiment['negative']}%") print(f"Neutral: {sentiment['neutral']}%") ``` #### Direct Database Access ```python import mysql.connector from datetime import datetime, timedelta conn = mysql.connector.connect( host=os.getenv('MYSQL_HOST'), user=os.getenv('MYSQL_USER'), password=os.getenv('MYSQL_PASSWORD'), database=os.getenv('MYSQL_DATABASE') ) cursor = conn.cursor(dictionary=True) # Get recent hot searches cursor.execute(""" SELECT platform, title, heat_value, url, timestamp FROM hot_searches WHERE timestamp >= %s ORDER BY rank ASC LIMIT 50 """, (datetime.now() - timedelta(hours=24),)) hot_topics = cursor.fetchall() for topic in hot_topics: print(f"[{topic['platform']}] {topic['title']} - {topic['heat_value']}") ``` #### Setting Up Push Notifications ```python from hotsearch_analysis_agent.push_service import PushService # Initialize push service push_service = PushService() # Create push task task_config = { 'name': 'AI Tech Trending Monitor', 'keywords': ['人工智能', '大模型', 'AI技术'], 'platforms': ['weibo', 'bilibili', 'zhihu'], 'schedule': '0 9,18 * * *', # Twice daily at 9 AM and 6 PM 'channels': ['email', 'wechat_work', 'telegram'], 'analysis_depth': 'detailed', # 'summary' or 'detailed' 'min_heat_threshold': 100000 } push_service.create_task(task_config) # Test push notification push_service.test_push( channel='email', subject='Test: AI Trending Report', content='This is a test notification.' ) ``` ### Crawler Management ```python from hotsearchcrawler.crawler_manager import CrawlerManager manager = CrawlerManager() # Start all crawlers manager.start_all() # Start specific platform manager.start_spider('weibo_spider') # Stop all crawlers manager.stop_all() # Get crawler status status = manager.get_status() print(f"Active crawlers: {status['active']}") print(f"Items scraped: {status['items_count']}") ``` ## Common Patterns ### Pattern 1: Daily Hot Topic Report ```python from hotsearch_analysis_agent.report_generator import ReportGenerator from datetime import datetime generator = ReportGenerator() # Generate daily report report = generator.generate_daily_report( date=datetime.now(), topics=['科技', '财经', '国际'], include_sentiment=True, include_clustering=True, output_format='markdown' ) # Save report with open(f"report_{datetime.now().strftime('%Y%m%d')}.md", 'w', encoding='utf-8') as f: f.write(report) # Auto-push report generator.push_report(report, channels=['email', 'wechat_work']) ``` ### Pattern 2: Real-Time Keyword Monitoring ```python from hotsearch_analysis_agent.monitor import KeywordMonitor import time monitor = KeywordMonitor() # Define alert keywords critical_keywords = ['安全事故', '数据泄露', '产品召回'] monitor.add_keywords(critical_keywords) # Start monitoring while True: alerts = monitor.check_new_mentions() for alert in alerts: print(f"ALERT: {alert['keyword']} mentioned in {alert['platform']}") print(f"Title: {alert['title']}") print(f"Heat: {alert['heat_value']}") print(f"URL: {alert['url']}") # Immediate push notification monitor.push_alert(alert, priority='high') time.sleep(300) # Check every 5 minutes ``` ### Pattern 3: Multi-Platform Topic Correlation ```python from hotsearch_analysis_agent.correlator import TopicCorrelator correlator = TopicCorrelator() # Find correlated topics across platforms topic_keyword = "芯片技术" correlation = correlator.find_cross_platform_correlation( keyword=topic_keyword, platforms=['weibo', 'zhihu', 'toutiao', 'bilibili'], time_window_hours=48 ) print(f"Topic: {topic_keyword}") print(f"Total mentions: {correlation['total_mentions']}") print(f"Platform distribution: {correlation['platform_dist']}") print(f"Peak time: {correlation['peak_timestamp']}") print(f"Related topics: {', '.join(correlation['related_topics'])}") ``` ### Pattern 4: Sentiment Trend Analysis ```python from hotsearch_analysis_agent.sentiment_tracker import SentimentTracker import matplotlib.pyplot as plt tracker = SentimentTracker() # Track sentiment over time sentiment_history = tracker.track_sentiment( keyword="新能源汽车", days=30, platforms=['weibo', 'zhihu'] ) # Visualize trend dates = [s['date'] for s in sentiment_history] positive = [s['positive'] for s in sentiment_history] negative = [s['negative'] for s in sentiment_history] plt.figure(figsize=(12, 6)) plt.plot(dates, positive, label='Positive', color='green') plt.plot(dates, negative, label='Negative', color='red') plt.xlabel('Date') plt.ylabel('Sentiment Score (%)') plt.title('Sentiment Trend: 新能源汽车') plt.legend() plt.savefig('sentiment_trend.png') ``` ## Testing ### Test Individual Components ```bash # Test crawler functionality python runspider-test.py # Test push notification python test_push_task.py # Test LLM analysis python -m hotsearch_analysis_agent.test_analysis ``` ### Sample Test Script ```python # test_system.py import os from dotenv import load_dotenv from hotsearch_analysis_agent.agent import OpinionAnalysisAgent load_dotenv() def test_query(): agent = OpinionAnalysisAgent() result = agent.query("What are the top 5 trending topics today?") assert result is not None assert 'analysis' in result print("✓ Query test passed") def test_clustering(): agent = OpinionAnalysisAgent()