data-scraper-agent

name	data-scraper-agent
description	Build an automated data collection agent for any public source. Use when the user wants to scrape, monitor, or track public data (jobs, prices, news) on a schedule with LLM enrichment and storage to Notion/Sheets/Supabase.
origin	community

Data Scraper Agent

Build a production-ready, AI-powered data collection agent for any public data source. Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time.

Stack: Python / Gemini Flash (free) / GitHub Actions (free) / Notion / Sheets / Supabase

When to Activate

• User wants to scrape or monitor any public website or API
• User says "build a bot that checks...", "monitor X for me", "collect data from..."
• User wants to track jobs, prices, news, repos, sports scores, events, listings
• User asks how to automate data collection without paying for hosting
• User wants an agent that gets smarter over time based on their decisions

Core Concepts

The Three Layers

Every data scraper agent has three layers:

COLLECT -> ENRICH -> STORE
  |           |        |
Scraper    AI (LLM)  Database
runs on    scores/   Notion /
schedule   summarises Sheets /
           & classifies Supabase

Free Stack

Layer	Tool	Why
Scraping	requests + BeautifulSoup	No cost, covers 80% of public sites
JS-rendered sites	playwright (free)	When HTML scraping fails
AI enrichment	Gemini Flash via REST API	500 req/day, 1M tokens/day -- free
Storage	Notion API	Free tier, great UI for review
Schedule	GitHub Actions cron	Free for public repos
Learning	JSON feedback file in repo	Zero infra, persists in git

AI Model Fallback Chain

Build agents to auto-fallback across Gemini models on quota exhaustion:

gemini-2.0-flash-lite (30 RPM) ->
gemini-2.0-flash (15 RPM) ->
gemini-2.5-flash (10 RPM) ->
gemini-flash-lite-latest (fallback)

Batch API Calls for Efficiency

Never call the LLM once per item. Always batch:

# BAD: 33 API calls for 33 items
for item in items:
    result = call_ai(item)  # 33 calls -> hits rate limit

# GOOD: 7 API calls for 33 items (batch size 5)
for batch in chunks(items, size=5):
    results = call_ai(batch)  # 7 calls -> stays within free tier

Workflow

Step 1: Understand the Goal

Ask the user:

1. What to collect: "What data source? URL / API / RSS / public endpoint?"
2. What to extract: "What fields matter? Title, price, URL, date, score?"
3. How to store: "Where should results go? Notion, Google Sheets, Supabase, or local file?"
4. How to enrich: "Do you want AI to score, summarise, classify, or match each item?"
5. Frequency: "How often should it run? Every hour, daily, weekly?"

Common examples to prompt:

• Job boards -> score relevance to resume
• Product prices -> alert on drops
• GitHub repos -> summarise new releases
• News feeds -> classify by topic + sentiment
• Sports results -> extract stats to tracker
• Events calendar -> filter by interest

Step 2: Design the Agent Architecture

my-agent/
├── config.yaml              # User customises this
├── profile/context.md        # User context for AI
├── scraper/
│   ├── main.py              # Orchestrator
│   ├── filters.py           # Rule-based pre-filter
│   └── sources/             # One file per data source
├── ai/
│   ├── client.py            # Gemini REST client with fallback
│   ├── pipeline.py          # Batch AI analysis
│   └── memory.py            # Learn from user feedback
├── storage/
│   └── notion_sync.py       # Or sheets_sync.py / supabase_sync.py
├── data/feedback.json        # User decision history
├── .env.example
├── requirements.txt
└── .github/workflows/scraper.yml

Steps 3-10: Implementation

See implementation-templates.md for complete code templates covering:

• Scraper source template (Step 3)
• Gemini AI client with fallback (Step 4)
• Batch AI pipeline (Step 5)
• Feedback learning system (Step 6)
• Storage layer -- Notion example (Step 7)
• Orchestrator main.py (Step 8)
• GitHub Actions workflow (Step 9)
• config.yaml template (Step 10)

Common Scraping Patterns

See scraping-patterns.md for detailed code for each pattern:

Pattern	When to Use
REST API	Public API with JSON responses (easiest)
HTML Scraping	Static HTML pages with structured markup
RSS Feed	Blogs, news sites, podcast feeds
Paginated API	APIs that return results across multiple pages
JS-Rendered Pages	SPAs or dynamic content requiring Playwright

Anti-Patterns to Avoid

Anti-pattern	Problem	Fix
One LLM call per item	Hits rate limits instantly	Batch 5 items per call
Hardcoded keywords in code	Not reusable	Move all config to config.yaml
Scraping without rate limit	IP ban	Add time.sleep(1) between requests
Storing secrets in code	Security risk	Always use .env + GitHub Secrets
No deduplication	Duplicate rows pile up	Always check URL before pushing
Ignoring robots.txt	Legal/ethical risk	Respect crawl rules; use public APIs when available
JS-rendered sites with requests	Empty response	Use Playwright or look for the underlying API
maxOutputTokens too low	Truncated JSON, parse error	Use 2048+ for batch responses

Free Tier Limits Reference

Service	Free Limit	Typical Usage
Gemini Flash Lite	30 RPM, 1500 RPD	~56 req/day at 3-hr intervals
Gemini 2.0 Flash	15 RPM, 1500 RPD	Good fallback
Gemini 2.5 Flash	10 RPM, 500 RPD	Use sparingly
GitHub Actions	Unlimited (public repos)	~20 min/day
Notion API	Unlimited	~200 writes/day
Supabase	500MB DB, 2GB transfer	Fine for most agents
Google Sheets API	300 req/min	Works for small agents

Quality Checklist

Before marking the agent complete:

• config.yaml controls all user-facing settings -- no hardcoded values
• profile/context.md holds user-specific context for AI matching
• Deduplication by URL before every storage push
• Gemini client has model fallback chain (4 models)
• Batch size <= 5 items per API call
• maxOutputTokens >= 2048
• .env is in .gitignore
• .env.example provided for onboarding
• setup.py creates DB schema on first run
• enrich_existing.py backfills AI scores on old rows
• GitHub Actions workflow commits feedback.json after each run
• README covers: setup in < 5 minutes, required secrets, customisation

Real-World Examples

"Build me an agent that monitors Hacker News for AI startup funding news"
"Scrape product prices from 3 e-commerce sites and alert when they drop"
"Track new GitHub repos tagged with 'llm' or 'agents' -- summarise each one"
"Collect Chief of Staff job listings from LinkedIn and Cutshort into Notion"
"Monitor a subreddit for posts mentioning my company -- classify sentiment"
"Scrape new academic papers from arXiv on a topic I care about daily"
"Track sports fixture results and keep a running table in Google Sheets"
"Build a real estate listing watcher -- alert on new properties under 1 Cr"

Reference Files

• implementation-templates.md -- Complete code templates for all agent components (Steps 3-10)
• scraping-patterns.md -- Detailed scraping pattern code and requirements template