scraper

name	scraper
description	Add, debug, or modify price scrapers for e-commerce sites (Shopee, Lazada, Tiki, etc.). Use when working with scraper configs or pullData modules.

Scraper Skill

Work with PriceTrack's price scrapers — add new sites, debug existing ones, or modify scraping logic.

Architecture

Scrapers are split into two parts:

1. Config (functions/config/<site>.js) — Site-specific selectors and URL patterns. Exports an object with:

   • urlPattern — regex to match product URLs
   • selectors — CSS selectors for price, title, image, etc.
   • transform — optional data transformation functions

2. Module (functions/modules/pullData.js) — Generic pull logic that reads configs and fetches/scrapes product pages using:

   • Puppeteer-core + @sparticuz/chromium for JS-rendered pages (serverless Chromium)
   • jsdom for static HTML parsing
   • axios for simple HTTP fetches

Adding a New Scraper

1. Create functions/config/<site>.js with the site's URL pattern and selectors
2. Register it in the config index (check how existing sites are imported)
3. Test with: firebase functions:shell → pullData({url: '<product-url>'})
4. Verify the scraped data structure matches what modules/pullData.js expects

Debugging

• Check Firebase Functions logs: firebase functions:log
• Puppeteer scrapers run headless on serverless Chromium — no GPU, limited memory
• Some sites require specific headers or cookie consent handling
• @sparticuz/chromium has size limits; check if the binary is up to date

Key Files

• functions/config/ — per-site scraper configurations
• functions/modules/pullData.js — main scraping orchestration
• functions/modules/updateInfo.js — product info updates
• functions/utils/fetch.js — HTTP fetch utilities