| name | data-scraper-agent |
| description | Build an automated data collection agent for any public source. Use when the user wants to scrape, monitor, or track public data (jobs, prices, news) on a schedule with LLM enrichment and storage to Notion/Sheets/Supabase. |
| origin | community |
Data Scraper Agent
Build a production-ready, AI-powered data collection agent for any public data source.
Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time.
Stack: Python / Gemini Flash (free) / GitHub Actions (free) / Notion / Sheets / Supabase
When to Activate
- User wants to scrape or monitor any public website or API
- User says "build a bot that checks...", "monitor X for me", "collect data from..."
- User wants to track jobs, prices, news, repos, sports scores, events, listings
- User asks how to automate data collection without paying for hosting
- User wants an agent that gets smarter over time based on their decisions
Core Concepts
The Three Layers
Every data scraper agent has three layers:
COLLECT -> ENRICH -> STORE
| | |
Scraper AI (LLM) Database
runs on scores/ Notion /
schedule summarises Sheets /
& classifies Supabase
Free Stack
| Layer | Tool | Why |
|---|
| Scraping | requests + BeautifulSoup | No cost, covers 80% of public sites |
| JS-rendered sites | playwright (free) | When HTML scraping fails |
| AI enrichment | Gemini Flash via REST API | 500 req/day, 1M tokens/day -- free |
| Storage | Notion API | Free tier, great UI for review |
| Schedule | GitHub Actions cron | Free for public repos |
| Learning | JSON feedback file in repo | Zero infra, persists in git |
AI Model Fallback Chain
Build agents to auto-fallback across Gemini models on quota exhaustion:
gemini-2.0-flash-lite (30 RPM) ->
gemini-2.0-flash (15 RPM) ->
gemini-2.5-flash (10 RPM) ->
gemini-flash-lite-latest (fallback)
Batch API Calls for Efficiency
Never call the LLM once per item. Always batch:
for item in items:
result = call_ai(item)
for batch in chunks(items, size=5):
results = call_ai(batch)
Workflow
Step 1: Understand the Goal
Ask the user:
- What to collect: "What data source? URL / API / RSS / public endpoint?"
- What to extract: "What fields matter? Title, price, URL, date, score?"
- How to store: "Where should results go? Notion, Google Sheets, Supabase, or local file?"
- How to enrich: "Do you want AI to score, summarise, classify, or match each item?"
- Frequency: "How often should it run? Every hour, daily, weekly?"
Common examples to prompt:
- Job boards -> score relevance to resume
- Product prices -> alert on drops
- GitHub repos -> summarise new releases
- News feeds -> classify by topic + sentiment
- Sports results -> extract stats to tracker
- Events calendar -> filter by interest
Step 2: Design the Agent Architecture
my-agent/
├── config.yaml # User customises this
├── profile/context.md # User context for AI
├── scraper/
│ ├── main.py # Orchestrator
│ ├── filters.py # Rule-based pre-filter
│ └── sources/ # One file per data source
├── ai/
│ ├── client.py # Gemini REST client with fallback
│ ├── pipeline.py # Batch AI analysis
│ └── memory.py # Learn from user feedback
├── storage/
│ └── notion_sync.py # Or sheets_sync.py / supabase_sync.py
├── data/feedback.json # User decision history
├── .env.example
├── requirements.txt
└── .github/workflows/scraper.yml
Steps 3-10: Implementation
See implementation-templates.md for complete code templates covering:
- Scraper source template (Step 3)
- Gemini AI client with fallback (Step 4)
- Batch AI pipeline (Step 5)
- Feedback learning system (Step 6)
- Storage layer -- Notion example (Step 7)
- Orchestrator main.py (Step 8)
- GitHub Actions workflow (Step 9)
- config.yaml template (Step 10)
Common Scraping Patterns
See scraping-patterns.md for detailed code for each pattern:
| Pattern | When to Use |
|---|
| REST API | Public API with JSON responses (easiest) |
| HTML Scraping | Static HTML pages with structured markup |
| RSS Feed | Blogs, news sites, podcast feeds |
| Paginated API | APIs that return results across multiple pages |
| JS-Rendered Pages | SPAs or dynamic content requiring Playwright |
Anti-Patterns to Avoid
| Anti-pattern | Problem | Fix |
|---|
| One LLM call per item | Hits rate limits instantly | Batch 5 items per call |
| Hardcoded keywords in code | Not reusable | Move all config to config.yaml |
| Scraping without rate limit | IP ban | Add time.sleep(1) between requests |
| Storing secrets in code | Security risk | Always use .env + GitHub Secrets |
| No deduplication | Duplicate rows pile up | Always check URL before pushing |
Ignoring robots.txt | Legal/ethical risk | Respect crawl rules; use public APIs when available |
JS-rendered sites with requests | Empty response | Use Playwright or look for the underlying API |
maxOutputTokens too low | Truncated JSON, parse error | Use 2048+ for batch responses |
Free Tier Limits Reference
| Service | Free Limit | Typical Usage |
|---|
| Gemini Flash Lite | 30 RPM, 1500 RPD | ~56 req/day at 3-hr intervals |
| Gemini 2.0 Flash | 15 RPM, 1500 RPD | Good fallback |
| Gemini 2.5 Flash | 10 RPM, 500 RPD | Use sparingly |
| GitHub Actions | Unlimited (public repos) | ~20 min/day |
| Notion API | Unlimited | ~200 writes/day |
| Supabase | 500MB DB, 2GB transfer | Fine for most agents |
| Google Sheets API | 300 req/min | Works for small agents |
Quality Checklist
Before marking the agent complete:
Real-World Examples
"Build me an agent that monitors Hacker News for AI startup funding news"
"Scrape product prices from 3 e-commerce sites and alert when they drop"
"Track new GitHub repos tagged with 'llm' or 'agents' -- summarise each one"
"Collect Chief of Staff job listings from LinkedIn and Cutshort into Notion"
"Monitor a subreddit for posts mentioning my company -- classify sentiment"
"Scrape new academic papers from arXiv on a topic I care about daily"
"Track sports fixture results and keep a running table in Google Sheets"
"Build a real estate listing watcher -- alert on new properties under 1 Cr"
Reference Files
