| name | scraper-debugger |
| description | Use this skill when a scraper workflow is failing in GitHub Actions, when a solver is returning 0 records, or when debugging any runtime error in the scraping pipeline. Contains diagnostic procedures, common failure modes, and fix patterns.
|
Scraper Debugger
This skill contains diagnostic procedures for debugging failing arrest scrapers.
Diagnostic Procedure
When a scraper is failing, follow this checklist in order:
Step 1: Identify the Failure
curl -s "https://api.github.com/repos/Shamrock2245/swfl-arrest-scrapers/actions/runs?per_page=20" \
| python3 -c "import json,sys; [print(f'{r[\"conclusion\"]:>10} | {r[\"name\"]:30} | {r[\"created_at\"]}') for r in json.load(sys.stdin)['workflow_runs']]"
Step 2: Check the Solver Function Name
The universal runner looks for scrape_{county_name}() in counties/{name}/solver.py.
grep -n "def scrape" counties/{name}/solver.py
The function name MUST match the folder name.
Step 3: Check the Solver Return Type
The solver MUST return a list, not print() JSON.
grep -n "print\|return" counties/{name}/solver.py | tail -20
Step 4: Check the Website
Verify the target website is still accessible and the HTML structure hasn't changed:
curl -s -o /dev/null -w "%{http_code}" "https://target-url.com"
Step 5: Check for Argument Mismatches
If using an old-style runner (python_scrapers/scrapers/run_*.py), verify:
- The runner's subprocess call passes args the solver's argparse expects
- Named args (
--start) vs positional args
Step 6: Check for Missing Dependencies
pip list | grep -i "drissionpage\|playwright\|curl_cffi\|beautifulsoup4"
Common Failure Modes
1. "No scrape function found"
Cause: Solver exports scrape_county() or scrape() instead of scrape_{county_name}().
Fix: Rename the function to match the county directory name.
2. Runner returns None → sys.exit(1)
Cause: Solver raises an exception or returns None instead of a list.
Fix: Wrap solver in try/except that returns [] on failure.
3. "error: unrecognized arguments"
Cause: Old-style runner passes wrong argument format to solver's argparse.
Fix: Check solver's argparse and match the runner's subprocess call.
4. Detail pages return empty HTML
Cause: Website uses JavaScript rendering (React SPA). requests + BeautifulSoup gets an empty shell.
Fix: Either:
- Extract data from the listing page (which is server-rendered)
- Use DrissionPage or Playwright for browser-based scraping
5. Cloudflare challenge blocks scraper
Cause: Bot detection on target site. requests gets a 403 or challenge page.
Fix: Use curl_cffi (with impersonate) or DrissionPage (full browser).
6. DrissionPage/Chromium not available
Cause: Chromium binary not installed on runner (common on minimal Docker/CI images).
Fix: Add google-chrome-stable to the runner setup, or wrap import in try/except.
7. Google Sheets write fails
Cause: Service account credentials missing or expired.
Fix: Check GOOGLE_SERVICE_ACCOUNT_KEY_PATH env var and creds/service-account-key.json file.
Environment Variables
GOOGLE_SHEETS_ID=121z5R6Hpqur54GNPC8L26ccfDPLHTJc3_LU6G7IV_0E
GOOGLE_SERVICE_ACCOUNT_KEY_PATH=creds/service-account-key.json
SLACK_WEBHOOK_URL=<configured per environment>
TZ=America/New_York
County Status (Last Updated: 2026-04-14)
✅ Passing
Charlotte, DeSoto, Hendry, Hillsborough, Manatee, Sarasota, Orange, Palm Beach, Seminole, Pinellas, Polk, Brevard, Martin, Lake
🔧 Recently Fixed
- Collier: Function name mismatch + print instead of return
- Osceola: Argument mismatch in subprocess call
- Highlands: DrissionPage import guard added
- Indian River: Rewritten to extract from listing page (SPA detail pages)
