retail-provider-fix

name	retail-provider-fix
description	Diagnose and fix retail scraping failures for individual dealers — wrong prices, OOS, site structure changes.

Retail Provider Fix

Diagnose and fix scraping failures for individual dealers in the StakTrakr retail pipeline.

Where things live (current as of 2026-04)

Piece	Path / Location
Extraction code	devops/pollers/shared/price-extract.js
Per-provider config	PROVIDER_CONFIG in price-extract.js
CF-clearance / Byparr sidecar	devops/pollers/shared/cf-clearance.js
Goldback scraper	devops/pollers/shared/goldback-scraper.js
Fly.io app	devops/pollers/remote-poller/ (app staktrakr)
Home poller (LXC)	devops/pollers/home-poller/
Home dashboard / provider editor	http://<home-poller-lxc>:3010
Cron source of truth	devops/pollers/home-poller/docker-entrypoint.sh
providers.json source of truth	sqld tables provider_coins + provider_vendors — regenerated by export-providers-json.js

Critical topology (STAK-478):

• Home poller (LXC) is the sole retail + goldback scraper. Retail cron 30 * * * * (hourly at :30), goldback 5 16 * * * daily.
• Fly.io staktrakr is a thin publisher. RETAIL_ENABLED=0, GOLDBACK_ENABLED=0. Fly only runs spot, publish, serve.js. Do not debug retail on Fly.
• Both pollers consume the same PROVIDER_CONFIG via shared/price-extract.js.

Step 1 — Live diagnostic against the home poller

Scrape the failing URL directly through the home poller's Firecrawl to see what the extractor actually sees. Get the URL from the home dashboard provider editor (:3010) for the failing coin/vendor row.

# From the home LXC (or over Tailscale):
docker exec -it staktrakr-home-poller curl -s -X POST http://firecrawl-api:3002/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"PROVIDER_URL_HERE","formats":["markdown"],"waitFor":6000,"onlyMainContent":false}' \
  | python3 -c "import sys,json; d=json.load(sys.stdin); \
      md=d.get('data',{}).get('markdown',''); \
      w=d.get('data',{}).get('warning',''); \
      print('WARNING:',w); print('LEN:',len(md)); print(md[:5000])"

Step 2 — Interpret the output

Symptom	Meaning	Fix category
warning: waitFor not supported	Firecrawl image too old to honor waitFor	Upgrade Firecrawl image OR set phase: "phase0" (Playwright-direct renders JS natively)
Markdown is only Loading...	Product table is JS-rendered	Switch phase to cf-clearance-first or phase0
Metadata URL ≠ sourceURL (homepage redirect)	Bot detection (Cloudflare)	phase: "cf-clearance-first" + cf_clearance_fallback: true
Spot prices appear before product price	MARKDOWN_HEADER_SKIP_PATTERNS not matching	Update header-skip regex for provider
fractional_weight — 1/4 oz	Nav/menu fractional link matched before product	Set fractionalExempt: true OR tighten MARKDOWN_CUTOFF_PATTERNS
page loaded, no price	Page rendered but extractPrice regex missed	Update extraction logic for provider
out_of_stock — PRE-ORDER	Item actually on pre-order	Fine unless provider is in PREORDER_TOLERANT_PROVIDERS

Step 3 — Config map in price-extract.js

All runtime behavior is driven by a single table, PROVIDER_CONFIG. Look up by provider id, not by line number.

Symbol	Purpose
PROVIDER_CONFIG[id]	Per-provider overrides: phase, waitFor, waitUntil, waitAfter, timeout, onlyMainContent, retryOn408, fractionalExempt, cf_clearance_fallback, proxy
PROVIDER_DEFAULTS	Defaults merged into every provider
providerCfg(id)	Helper — returns merged config for a provider id
MARKDOWN_HEADER_SKIP_PATTERNS	Per-provider regex — cuts everything BEFORE the match
MARKDOWN_CUTOFF_PATTERNS	Per-provider regex array — cuts everything AFTER the first match
PREORDER_TOLERANT_PROVIDERS	Providers where PRE-ORDER does NOT imply OOS
USES_AS_LOW_AS	Providers that use "As Low As" as primary price (currently empty)
SLOW_PROVIDERS / FIRECRAWL_PREFERRED_PROVIDERS / PLAYWRIGHT_ONLY_PROVIDERS	Back-compat shims auto-derived from PROVIDER_CONFIG. Do not edit directly — change PROVIDER_CONFIG instead.

Never edit providers.json by hand. It is regenerated from sqld every 5 min via export-providers-json.js. Edit URLs/selectors through the home dashboard provider editor at :3010, which calls updateVendorUrl(client, coinSlug, vendorId, url) in provider-db.js.

Step 4 — Phase routing

phase on each provider selects the scrape strategy. This is the primary knob — flip it before chasing regex/header changes.

phase	Order
phase0 (default)	Playwright-direct → Firecrawl → CF-clearance fallback
firecrawl	Skip Phase 0, Firecrawl → CF-clearance fallback
cf-clearance-first	Byparr/CF-clearance sidecar → Firecrawl fallback

Current assignments in code:

• apmex, monumentmetals, herobullion, gainesvillecoins → firecrawl (HTML pipe tables lose structure via innerText)
• jmbullion, bullionexchanges → cf-clearance-first + cf_clearance_fallback: true (CF protected)
• sdbullion, summitmetals, goldback → phase0 defaults

Step 5 — CF-protected vendors (JMBullion, BullionExchanges)

Current solution is the Byparr sidecar reached via phase: "cf-clearance-first" and cf_clearance_fallback: true. This replaces the old residential-proxy / Tailscale-exit-node approach.

• Byparr fetches the page with a valid cf_clearance cookie and returns the HTML.
• If Byparr has no usable HTML, code launches Playwright with the cf_clearance cookie attached.
• Fly-based BullionExchanges is ~93% on Fly's own IP (no proxy needed); home poller uses its residential IP.

If CF bypass stops working:

1. docker logs staktrakr-byparr on the home LXC — confirm sidecar up.
2. Check shared/cf-clearance.js — cookie cache, retry count, timeout.
3. Verify retryOn408 and timeout are generous enough (bullionexchanges currently 70_000).

Step 6 — Dry run a single vendor

Home poller (primary path for retail):

# On the home LXC
docker exec -it staktrakr-home-poller bash
cd /app
DATA_DIR=/data \
FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase PROVIDERS=jmbullion \
DRY_RUN=1 node price-extract.js

# Multi-provider
DATA_DIR=/data \
FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase,age PROVIDERS=jmbullion,bullionexchanges \
DRY_RUN=1 node price-extract.js

Fly (only if RETAIL_ENABLED=1 is being re-enabled or validated):

fly ssh console -a staktrakr
cd /app
DATA_DIR=/data/staktrakr-api-export/data \
FIRECRAWL_BASE_URL=http://localhost:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase PROVIDERS=jmbullion \
DRY_RUN=1 node price-extract.js

Success: ✓ jmbullion: $XXX.XX (cf-clearance) / (firecrawl) / (playwright).

Step 7 — Ship the fix

• Code change in shared/price-extract.js or shared/cf-clearance.js — worktree → PR to dev. Home poller rebuilds its image and restarts to pick it up. Fly picks it up on next fly deploy -a staktrakr.
• URL / selector change — edit in home dashboard provider editor (:3010). Writes to sqld. export-providers-json.js regenerates providers.json within 5 min and publishes on next Fly run-publish cycle (8,23,38,53 * * * *).
• Threshold / cron tweak — edit devops/pollers/home-poller/docker-entrypoint.sh (authoritative) and/or devops/pollers/remote-poller/docker-entrypoint.sh.

Gap-fill / FindBullionPrices (FBP)

FBP is now a gap-fill pass, not a primary provider. devops/pollers/home-poller/run-fbp.sh runs with PATCH_GAPS=1, queries sqld for today's failed vendors, scrapes FBP for those specific coins, writes recovered prices. There is no longer a FBP_DEALER_NAME_MAP in price-extract.js — dealer resolution happens via FBP's own JSON joined against sqld provider_vendors.

Common provider failure patterns

JMBullion — fractional_weight

• Cause: mega-menu lists fractional coins on every product page; pre-render HTML trips the fractional check before full render.
• Fix: confirm PROVIDER_CONFIG.jmbullion.fractionalExempt = true and phase: "cf-clearance-first". Verify Byparr sidecar is up.

BullionExchanges — page loaded, no price

• Cause: React pricing grid needs ~12–15s to hydrate; CF block redirects Firecrawl to homepage.
• Fix: phase: "cf-clearance-first", waitFor: 15_000, timeout: 70_000, fractionalExempt: true. Confirm Byparr cf_clearance cookie is fresh.

Monument Metals — out_of_stock — PRE-ORDER

• Cause: Legitimate pre-order (common Jan–Mar for new-year mintings).
• Fix: Swap URL to an in-stock coin via the dashboard, or accept until inventory arrives. monumentmetals is in PREORDER_TOLERANT_PROVIDERS.

APMEX / Hero / Gainesville — pipe tables

• Cause: Pipe-delimited HTML tables — Playwright-direct innerText strips the pipes and firstTableRowFirstPrice() returns null, so firstInRangePriceProse() grabs a bulk "As Low As" price.
• Fix: phase: "firecrawl" (already set) — Firecrawl markdown preserves the | pipes. Don't regress to phase0.

SDBullion — wrong price from carousel

• Cause: sdbullion was previously in USES_AS_LOW_AS; "Add on Items" carousel bled through.
• Applied fix (2026-02-20): Removed from USES_AS_LOW_AS; added MARKDOWN_CUTOFF_PATTERNS.sdbullion.

Firecrawl engine quick reference

Self-hosted Firecrawl lives at http://firecrawl-api:3002 on home poller and http://localhost:3002 on Fly. Capabilities vary by image version — waitFor support in particular has changed across upgrades. Verify on the deployed version:

# Home
docker exec staktrakr-home-poller curl -s http://firecrawl-api:3002/health
docker exec firecrawl-api node -e "console.log(require('/opt/firecrawl/package.json').version)"

# Fly
fly ssh console -a staktrakr -C "node -e \"console.log(require('/opt/firecrawl/package.json').version)\""

If waitFor is silently ignored on the current image, PROVIDER_CONFIG's waitFor values are no-ops — use phase: "cf-clearance-first" or phase0 (Playwright-direct) for JS-heavy SPAs instead of relying on Firecrawl to wait.