Menu
Diagnose and fix retail scraping failures for individual dealers — wrong prices, OOS, site structure changes.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Use when starting a new patch session and needing to pick a Plane issue to work on before claiming a version lock and creating a worktree.
Use when starting a new patch session and needing to pick a Plane issue to work on before claiming a version lock and creating a worktree.
Post-deploy verification for both Portainer (home) and Fly.io (cloud) environments. Runs environment-appropriate health checks, log inspection, env var verification, cron validation, and endpoint smoke tests. Covers the gap between "deploy succeeded" and "deploy actually works." Triggers on: "deploy verify", "post-deploy", "verify deploy", "smoke test deploy", "did the deploy work", "check the deploy", "/deploy-verify"
StakTrakr-specific Phase 1 version bump — edits 6 files, trims What's New, delegates sw.js to pre-commit hook.
API feeds, pollers, feed thresholds, data paths, StakTrakrApi structure, Fly.io and health badge issues.
Ship dev→main — PR, resolve threads, GitHub Release. Only on explicit user "ready to ship".
| name | retail-provider-fix |
| description | Diagnose and fix retail scraping failures for individual dealers — wrong prices, OOS, site structure changes. |
Diagnose and fix scraping failures for individual dealers in the StakTrakr retail pipeline.
| Piece | Path / Location |
|---|---|
| Extraction code | devops/pollers/shared/price-extract.js |
| Per-provider config | PROVIDER_CONFIG in price-extract.js |
| CF-clearance / Byparr sidecar | devops/pollers/shared/cf-clearance.js |
| Goldback scraper | devops/pollers/shared/goldback-scraper.js |
| Fly.io app | devops/pollers/remote-poller/ (app staktrakr) |
| Home poller (LXC) | devops/pollers/home-poller/ |
| Home dashboard / provider editor | http://<home-poller-lxc>:3010 |
| Cron source of truth | devops/pollers/home-poller/docker-entrypoint.sh |
| providers.json source of truth | sqld tables provider_coins + provider_vendors — regenerated by export-providers-json.js |
Critical topology (STAK-478):
30 * * * * (hourly at :30), goldback 5 16 * * * daily.staktrakr is a thin publisher. RETAIL_ENABLED=0, GOLDBACK_ENABLED=0. Fly only runs spot, publish, serve.js. Do not debug retail on Fly.PROVIDER_CONFIG via shared/price-extract.js.Scrape the failing URL directly through the home poller's Firecrawl to see what the extractor actually sees. Get the URL from the home dashboard provider editor (:3010) for the failing coin/vendor row.
# From the home LXC (or over Tailscale):
docker exec -it staktrakr-home-poller curl -s -X POST http://firecrawl-api:3002/v1/scrape \
-H 'Content-Type: application/json' \
-d '{"url":"PROVIDER_URL_HERE","formats":["markdown"],"waitFor":6000,"onlyMainContent":false}' \
| python3 -c "import sys,json; d=json.load(sys.stdin); \
md=d.get('data',{}).get('markdown',''); \
w=d.get('data',{}).get('warning',''); \
print('WARNING:',w); print('LEN:',len(md)); print(md[:5000])"
| Symptom | Meaning | Fix category |
|---|---|---|
warning: waitFor not supported | Firecrawl image too old to honor waitFor | Upgrade Firecrawl image OR set phase: "phase0" (Playwright-direct renders JS natively) |
Markdown is only Loading... | Product table is JS-rendered | Switch phase to cf-clearance-first or phase0 |
| Metadata URL ≠ sourceURL (homepage redirect) | Bot detection (Cloudflare) | phase: "cf-clearance-first" + cf_clearance_fallback: true |
| Spot prices appear before product price | MARKDOWN_HEADER_SKIP_PATTERNS not matching | Update header-skip regex for provider |
fractional_weight — 1/4 oz | Nav/menu fractional link matched before product | Set fractionalExempt: true OR tighten MARKDOWN_CUTOFF_PATTERNS |
page loaded, no price | Page rendered but extractPrice regex missed | Update extraction logic for provider |
out_of_stock — PRE-ORDER | Item actually on pre-order | Fine unless provider is in PREORDER_TOLERANT_PROVIDERS |
price-extract.jsAll runtime behavior is driven by a single table, PROVIDER_CONFIG. Look up by provider id, not by line number.
| Symbol | Purpose |
|---|---|
PROVIDER_CONFIG[id] | Per-provider overrides: phase, waitFor, waitUntil, waitAfter, timeout, onlyMainContent, retryOn408, fractionalExempt, cf_clearance_fallback, proxy |
PROVIDER_DEFAULTS | Defaults merged into every provider |
providerCfg(id) | Helper — returns merged config for a provider id |
MARKDOWN_HEADER_SKIP_PATTERNS | Per-provider regex — cuts everything BEFORE the match |
MARKDOWN_CUTOFF_PATTERNS | Per-provider regex array — cuts everything AFTER the first match |
PREORDER_TOLERANT_PROVIDERS | Providers where PRE-ORDER does NOT imply OOS |
USES_AS_LOW_AS | Providers that use "As Low As" as primary price (currently empty) |
SLOW_PROVIDERS / FIRECRAWL_PREFERRED_PROVIDERS / PLAYWRIGHT_ONLY_PROVIDERS | Back-compat shims auto-derived from PROVIDER_CONFIG. Do not edit directly — change PROVIDER_CONFIG instead. |
Never edit providers.json by hand. It is regenerated from sqld every 5 min via export-providers-json.js. Edit URLs/selectors through the home dashboard provider editor at :3010, which calls updateVendorUrl(client, coinSlug, vendorId, url) in provider-db.js.
phase on each provider selects the scrape strategy. This is the primary knob — flip it before chasing regex/header changes.
phase | Order |
|---|---|
phase0 (default) | Playwright-direct → Firecrawl → CF-clearance fallback |
firecrawl | Skip Phase 0, Firecrawl → CF-clearance fallback |
cf-clearance-first | Byparr/CF-clearance sidecar → Firecrawl fallback |
Current assignments in code:
apmex, monumentmetals, herobullion, gainesvillecoins → firecrawl (HTML pipe tables lose structure via innerText)jmbullion, bullionexchanges → cf-clearance-first + cf_clearance_fallback: true (CF protected)sdbullion, summitmetals, goldback → phase0 defaultsCurrent solution is the Byparr sidecar reached via phase: "cf-clearance-first" and cf_clearance_fallback: true. This replaces the old residential-proxy / Tailscale-exit-node approach.
cf_clearance cookie and returns the HTML.cf_clearance cookie attached.If CF bypass stops working:
docker logs staktrakr-byparr on the home LXC — confirm sidecar up.shared/cf-clearance.js — cookie cache, retry count, timeout.retryOn408 and timeout are generous enough (bullionexchanges currently 70_000).Home poller (primary path for retail):
# On the home LXC
docker exec -it staktrakr-home-poller bash
cd /app
DATA_DIR=/data \
FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase PROVIDERS=jmbullion \
DRY_RUN=1 node price-extract.js
# Multi-provider
DATA_DIR=/data \
FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase,age PROVIDERS=jmbullion,bullionexchanges \
DRY_RUN=1 node price-extract.js
Fly (only if RETAIL_ENABLED=1 is being re-enabled or validated):
fly ssh console -a staktrakr
cd /app
DATA_DIR=/data/staktrakr-api-export/data \
FIRECRAWL_BASE_URL=http://localhost:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase PROVIDERS=jmbullion \
DRY_RUN=1 node price-extract.js
Success: ✓ jmbullion: $XXX.XX (cf-clearance) / (firecrawl) / (playwright).
shared/price-extract.js or shared/cf-clearance.js — worktree → PR to dev. Home poller rebuilds its image and restarts to pick it up. Fly picks it up on next fly deploy -a staktrakr.:3010). Writes to sqld. export-providers-json.js regenerates providers.json within 5 min and publishes on next Fly run-publish cycle (8,23,38,53 * * * *).devops/pollers/home-poller/docker-entrypoint.sh (authoritative) and/or devops/pollers/remote-poller/docker-entrypoint.sh.FBP is now a gap-fill pass, not a primary provider. devops/pollers/home-poller/run-fbp.sh runs with PATCH_GAPS=1, queries sqld for today's failed vendors, scrapes FBP for those specific coins, writes recovered prices. There is no longer a FBP_DEALER_NAME_MAP in price-extract.js — dealer resolution happens via FBP's own JSON joined against sqld provider_vendors.
fractional_weightPROVIDER_CONFIG.jmbullion.fractionalExempt = true and phase: "cf-clearance-first". Verify Byparr sidecar is up.page loaded, no pricephase: "cf-clearance-first", waitFor: 15_000, timeout: 70_000, fractionalExempt: true. Confirm Byparr cf_clearance cookie is fresh.out_of_stock — PRE-ORDERmonumentmetals is in PREORDER_TOLERANT_PROVIDERS.innerText strips the pipes and firstTableRowFirstPrice() returns null, so firstInRangePriceProse() grabs a bulk "As Low As" price.phase: "firecrawl" (already set) — Firecrawl markdown preserves the | pipes. Don't regress to phase0.sdbullion was previously in USES_AS_LOW_AS; "Add on Items" carousel bled through.USES_AS_LOW_AS; added MARKDOWN_CUTOFF_PATTERNS.sdbullion.Self-hosted Firecrawl lives at http://firecrawl-api:3002 on home poller and http://localhost:3002 on Fly. Capabilities vary by image version — waitFor support in particular has changed across upgrades. Verify on the deployed version:
# Home
docker exec staktrakr-home-poller curl -s http://firecrawl-api:3002/health
docker exec firecrawl-api node -e "console.log(require('/opt/firecrawl/package.json').version)"
# Fly
fly ssh console -a staktrakr -C "node -e \"console.log(require('/opt/firecrawl/package.json').version)\""
If waitFor is silently ignored on the current image, PROVIDER_CONFIG's waitFor values are no-ops — use phase: "cf-clearance-first" or phase0 (Playwright-direct) for JS-heavy SPAs instead of relying on Firecrawl to wait.