| name | retail-provider-fix |
| description | Diagnose and fix retail scraping failures for individual dealers — wrong prices, OOS, site structure changes. |
Retail Provider Fix
Diagnose and fix scraping failures for individual dealers in the StakTrakr retail pipeline.
Where things live (current as of 2026-04)
| Piece | Path / Location |
|---|
| Extraction code | devops/pollers/shared/price-extract.js |
| Per-provider config | PROVIDER_CONFIG in price-extract.js |
| CF-clearance / Byparr sidecar | devops/pollers/shared/cf-clearance.js |
| Goldback scraper | devops/pollers/shared/goldback-scraper.js |
| Fly.io app | devops/pollers/remote-poller/ (app staktrakr) |
| Home poller (LXC) | devops/pollers/home-poller/ |
| Home dashboard / provider editor | http://<home-poller-lxc>:3010 |
| Cron source of truth | devops/pollers/home-poller/docker-entrypoint.sh |
| providers.json source of truth | sqld tables provider_coins + provider_vendors — regenerated by export-providers-json.js |
Critical topology (STAK-478):
- Home poller (LXC) is the sole retail + goldback scraper. Retail cron
30 * * * * (hourly at :30), goldback 5 16 * * * daily.
- Fly.io
staktrakr is a thin publisher. RETAIL_ENABLED=0, GOLDBACK_ENABLED=0. Fly only runs spot, publish, serve.js. Do not debug retail on Fly.
- Both pollers consume the same
PROVIDER_CONFIG via shared/price-extract.js.
Step 1 — Live diagnostic against the home poller
Scrape the failing URL directly through the home poller's Firecrawl to see what the extractor actually sees. Get the URL from the home dashboard provider editor (:3010) for the failing coin/vendor row.
docker exec -it staktrakr-home-poller curl -s -X POST http://firecrawl-api:3002/v1/scrape \
-H 'Content-Type: application/json' \
-d '{"url":"PROVIDER_URL_HERE","formats":["markdown"],"waitFor":6000,"onlyMainContent":false}' \
| python3 -c "import sys,json; d=json.load(sys.stdin); \
md=d.get('data',{}).get('markdown',''); \
w=d.get('data',{}).get('warning',''); \
print('WARNING:',w); print('LEN:',len(md)); print(md[:5000])"
Step 2 — Interpret the output
| Symptom | Meaning | Fix category |
|---|
warning: waitFor not supported | Firecrawl image too old to honor waitFor | Upgrade Firecrawl image OR set phase: "phase0" (Playwright-direct renders JS natively) |
Markdown is only Loading... | Product table is JS-rendered | Switch phase to cf-clearance-first or phase0 |
| Metadata URL ≠ sourceURL (homepage redirect) | Bot detection (Cloudflare) | phase: "cf-clearance-first" + cf_clearance_fallback: true |
| Spot prices appear before product price | MARKDOWN_HEADER_SKIP_PATTERNS not matching | Update header-skip regex for provider |
fractional_weight — 1/4 oz | Nav/menu fractional link matched before product | Set fractionalExempt: true OR tighten MARKDOWN_CUTOFF_PATTERNS |
page loaded, no price | Page rendered but extractPrice regex missed | Update extraction logic for provider |
out_of_stock — PRE-ORDER | Item actually on pre-order | Fine unless provider is in PREORDER_TOLERANT_PROVIDERS |
Step 3 — Config map in price-extract.js
All runtime behavior is driven by a single table, PROVIDER_CONFIG. Look up by provider id, not by line number.
| Symbol | Purpose |
|---|
PROVIDER_CONFIG[id] | Per-provider overrides: phase, waitFor, waitUntil, waitAfter, timeout, onlyMainContent, retryOn408, fractionalExempt, cf_clearance_fallback, proxy |
PROVIDER_DEFAULTS | Defaults merged into every provider |
providerCfg(id) | Helper — returns merged config for a provider id |
MARKDOWN_HEADER_SKIP_PATTERNS | Per-provider regex — cuts everything BEFORE the match |
MARKDOWN_CUTOFF_PATTERNS | Per-provider regex array — cuts everything AFTER the first match |
PREORDER_TOLERANT_PROVIDERS | Providers where PRE-ORDER does NOT imply OOS |
USES_AS_LOW_AS | Providers that use "As Low As" as primary price (currently empty) |
SLOW_PROVIDERS / FIRECRAWL_PREFERRED_PROVIDERS / PLAYWRIGHT_ONLY_PROVIDERS | Back-compat shims auto-derived from PROVIDER_CONFIG. Do not edit directly — change PROVIDER_CONFIG instead. |
Never edit providers.json by hand. It is regenerated from sqld every 5 min via export-providers-json.js. Edit URLs/selectors through the home dashboard provider editor at :3010, which calls updateVendorUrl(client, coinSlug, vendorId, url) in provider-db.js.
Step 4 — Phase routing
phase on each provider selects the scrape strategy. This is the primary knob — flip it before chasing regex/header changes.
phase | Order |
|---|
phase0 (default) | Playwright-direct → Firecrawl → CF-clearance fallback |
firecrawl | Skip Phase 0, Firecrawl → CF-clearance fallback |
cf-clearance-first | Byparr/CF-clearance sidecar → Firecrawl fallback |
Current assignments in code:
apmex, monumentmetals, herobullion, gainesvillecoins → firecrawl (HTML pipe tables lose structure via innerText)jmbullion, bullionexchanges → cf-clearance-first + cf_clearance_fallback: true (CF protected)sdbullion, summitmetals, goldback → phase0 defaults
Step 5 — CF-protected vendors (JMBullion, BullionExchanges)
Current solution is the Byparr sidecar reached via phase: "cf-clearance-first" and cf_clearance_fallback: true. This replaces the old residential-proxy / Tailscale-exit-node approach.
- Byparr fetches the page with a valid
cf_clearance cookie and returns the HTML.
- If Byparr has no usable HTML, code launches Playwright with the
cf_clearance cookie attached.
- Fly-based BullionExchanges is ~93% on Fly's own IP (no proxy needed); home poller uses its residential IP.
If CF bypass stops working:
docker logs staktrakr-byparr on the home LXC — confirm sidecar up.- Check
shared/cf-clearance.js — cookie cache, retry count, timeout.
- Verify
retryOn408 and timeout are generous enough (bullionexchanges currently 70_000).
Step 6 — Dry run a single vendor
Home poller (primary path for retail):
docker exec -it staktrakr-home-poller bash
cd /app
DATA_DIR=/data \
FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase PROVIDERS=jmbullion \
DRY_RUN=1 node price-extract.js
DATA_DIR=/data \
FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase,age PROVIDERS=jmbullion,bullionexchanges \
DRY_RUN=1 node price-extract.js
Fly (only if RETAIL_ENABLED=1 is being re-enabled or validated):
fly ssh console -a staktrakr
cd /app
DATA_DIR=/data/staktrakr-api-export/data \
FIRECRAWL_BASE_URL=http://localhost:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase PROVIDERS=jmbullion \
DRY_RUN=1 node price-extract.js
Success: ✓ jmbullion: $XXX.XX (cf-clearance) / (firecrawl) / (playwright).
Step 7 — Ship the fix
- Code change in
shared/price-extract.js or shared/cf-clearance.js — worktree → PR to dev. Home poller rebuilds its image and restarts to pick it up. Fly picks it up on next fly deploy -a staktrakr.
- URL / selector change — edit in home dashboard provider editor (
:3010). Writes to sqld. export-providers-json.js regenerates providers.json within 5 min and publishes on next Fly run-publish cycle (8,23,38,53 * * * *).
- Threshold / cron tweak — edit
devops/pollers/home-poller/docker-entrypoint.sh (authoritative) and/or devops/pollers/remote-poller/docker-entrypoint.sh.
Gap-fill / FindBullionPrices (FBP)
FBP is now a gap-fill pass, not a primary provider. devops/pollers/home-poller/run-fbp.sh runs with PATCH_GAPS=1, queries sqld for today's failed vendors, scrapes FBP for those specific coins, writes recovered prices. There is no longer a FBP_DEALER_NAME_MAP in price-extract.js — dealer resolution happens via FBP's own JSON joined against sqld provider_vendors.
Common provider failure patterns
JMBullion — fractional_weight
- Cause: mega-menu lists fractional coins on every product page; pre-render HTML trips the fractional check before full render.
- Fix: confirm
PROVIDER_CONFIG.jmbullion.fractionalExempt = true and phase: "cf-clearance-first". Verify Byparr sidecar is up.
BullionExchanges — page loaded, no price
- Cause: React pricing grid needs ~12–15s to hydrate; CF block redirects Firecrawl to homepage.
- Fix:
phase: "cf-clearance-first", waitFor: 15_000, timeout: 70_000, fractionalExempt: true. Confirm Byparr cf_clearance cookie is fresh.
Monument Metals — out_of_stock — PRE-ORDER
- Cause: Legitimate pre-order (common Jan–Mar for new-year mintings).
- Fix: Swap URL to an in-stock coin via the dashboard, or accept until inventory arrives.
monumentmetals is in PREORDER_TOLERANT_PROVIDERS.
APMEX / Hero / Gainesville — pipe tables
- Cause: Pipe-delimited HTML tables — Playwright-direct
innerText strips the pipes and firstTableRowFirstPrice() returns null, so firstInRangePriceProse() grabs a bulk "As Low As" price.
- Fix:
phase: "firecrawl" (already set) — Firecrawl markdown preserves the | pipes. Don't regress to phase0.
SDBullion — wrong price from carousel
- Cause:
sdbullion was previously in USES_AS_LOW_AS; "Add on Items" carousel bled through.
- Applied fix (2026-02-20): Removed from
USES_AS_LOW_AS; added MARKDOWN_CUTOFF_PATTERNS.sdbullion.
Firecrawl engine quick reference
Self-hosted Firecrawl lives at http://firecrawl-api:3002 on home poller and http://localhost:3002 on Fly. Capabilities vary by image version — waitFor support in particular has changed across upgrades. Verify on the deployed version:
docker exec staktrakr-home-poller curl -s http://firecrawl-api:3002/health
docker exec firecrawl-api node -e "console.log(require('/opt/firecrawl/package.json').version)"
fly ssh console -a staktrakr -C "node -e \"console.log(require('/opt/firecrawl/package.json').version)\""
If waitFor is silently ignored on the current image, PROVIDER_CONFIG's waitFor values are no-ops — use phase: "cf-clearance-first" or phase0 (Playwright-direct) for JS-heavy SPAs instead of relying on Firecrawl to wait.
