Imported from lbruton/StakTrakr (.agents/skills/retail-provider-fix/SKILL.md). Install upstream with npx skills add lbruton/StakTrakr --skill retail-provider-fix. Copyright stays with the author.

Retail Provider Fix

Diagnose and fix scraping failures for individual dealers in the StakTrakr retail pipeline.

Where things live (current as of 2026-04)

Critical topology (STAK-478):

• Home poller (LXC) is the sole retail + goldback scraper. Retail cron 30 * * * * (hourly at :30), goldback 5 16 * * * daily.
• Fly.io staktrakr is a thin publisher. RETAIL_ENABLED=0, GOLDBACK_ENABLED=0. Fly only runs spot, publish, serve.js. Do not debug retail on Fly.
• Both pollers consume the same PROVIDER_CONFIG via shared/price-extract.js.

Step 1 — Live diagnostic against the home poller

Scrape the failing URL directly through the home poller's Firecrawl to see what the extractor actually sees. Get the URL from the home dashboard provider editor (:3010) for the failing coin/vendor row.

# From the home LXC (or over Tailscale):
docker exec -it staktrakr-home-poller curl -s -X POST http://firecrawl-api:3002/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"PROVIDER_URL_HERE","formats":["markdown"],"waitFor":6000,"onlyMainContent":false}' \
  | python3 -c "import sys,json; d=json.load(sys.stdin); \
      md=d.get('data',{}).get('markdown',''); \
      w=d.get('data',{}).get('warning',''); \
      print('WARNING:',w); print('LEN:',len(md)); print(md[:5000])"

Step 2 — Interpret the output

Step 3 — Config map in price-extract.js

All runtime behavior is driven by a single table, PROVIDER_CONFIG. Look up by provider id, not by line number.

Never edit providers.json by hand. It is regenerated from sqld every 5 min via export-providers-json.js. Edit URLs/selectors through the home dashboard provider editor at :3010, which calls updateVendorUrl(client, coinSlug, vendorId, url) in provider-db.js.

Step 4 — Phase routing

phase on each provider selects the scrape strategy. This is the primary knob — flip it before chasing regex/header changes.

Current assignments in code:

• apmex, monumentmetals, herobullion, gainesvillecoins → firecrawl (HTML pipe tables lose structure via innerText)
• jmbullion, bullionexchanges → cf-clearance-first + cf_clearance_fallback: true (CF protected)
• sdbullion, summitmetals, goldback → phase0 defaults

Step 5 — CF-protected vendors (JMBullion, BullionExchanges)

Current solution is the Byparr sidecar reached via phase: "cf-clearance-first" and cf_clearance_fallback: true. This replaces the old residential-proxy / Tailscale-exit-node approach.

• Byparr fetches the page with a valid cf_clearance cookie and returns the HTML.
• If Byparr has no usable HTML, code launches Playwright with the cf_clearance cookie attached.
• Fly-based BullionExchanges is ~93% on Fly's own IP (no proxy needed); home poller uses its residential IP.

If CF bypass stops working:

1. docker logs staktrakr-byparr on the home LXC — confirm sidecar up.
2. Check shared/cf-clearance.js — cookie cache, retry count, timeout.
3. Verify retryOn408 and timeout are generous enough (bullionexchanges currently 70_000).

Step 6 — Dry run a single vendor

Home poller (primary path for retail):

# On the home LXC
docker exec -it staktrakr-home-poller bash
cd /app
DATA_DIR=/data \
FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase PROVIDERS=jmbullion \
DRY_RUN=1 node price-extract.js

# Multi-provider
DATA_DIR=/data \
FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase,age PROVIDERS=jmbullion,bullionexchanges \
DRY_RUN=1 node price-extract.js

Fly (only if RETAIL_ENABLED=1 is being re-enabled or validated):

fly ssh console -a staktrakr
cd /app
DATA_DIR=/data/staktrakr-api-export/data \
FIRECRAWL_BASE_URL=http://localhost:3002 \
PLAYWRIGHT_LAUNCH=1 \
COINS=ase PROVIDERS=jmbullion \
DRY_RUN=1 node price-extract.js

Success: ✓ jmbullion: $XXX.XX (cf-clearance) / (firecrawl) / (playwright).

Step 7 — Ship the fix

• Code change in shared/price-extract.js or shared/cf-clearance.js — worktree → PR to dev. Home poller rebuilds its image and restarts to pick it up. Fly picks it up on next fly deploy -a staktrakr.
• URL / selector change — edit in home dashboard provider editor (:3010). Writes to sqld. export-providers-json.js regenerates providers.json within 5 min and publishes on next Fly run-publish cycle (8,23,38,53 * * * *).
• Threshold / cron tweak — edit devops/pollers/home-poller/docker-entrypoint.sh (authoritative) and/or devops/pollers/remote-poller/docker-entrypoint.sh.

Gap-fill / FindBullionPrices (FBP) — REMOVED

The FBP gap-fill pass no longer exists: nothing reads PATCH_GAPS, and extractFbpPrices() / FBP_DEALER_NAME_MAP are gone from price-extract.js. The dead home-poller/run-fbp.sh wrapper was deleted (2026-08-01). provider_coins.fbp_url remains as a live schema column; source: "fbp" rows in sqld are historical only.

Common provider failure patterns

JMBullion — fractional_weight

• Cause: mega-menu lists fractional coins on every product page; pre-render HTML trips the fractional check before full render.
• Fix: confirm PROVIDER_CONFIG.jmbullion.fractionalExempt = true and phase: "cf-clearance-first". Verify Byparr sidecar is up.

BullionExchanges — page loaded, no price

• Cause: React pricing grid needs ~12–15s to hydrate; CF block redirects Firecrawl to homepage.
• Fix: phase: "cf-clearance-first", waitFor: 15_000, timeout: 70_000, fractionalExempt: true. Confirm Byparr cf_clearance cookie is fresh.

Monument Metals — out_of_stock — PRE-ORDER

• Cause: Legitimate pre-order (common Jan–Mar for new-year mintings).
• Fix: Swap URL to an in-stock coin via the dashboard, or accept until inventory arrives. monumentmetals is in PREORDER_TOLERANT_PROVIDERS.

APMEX / Hero / Gainesville — pipe tables

• Cause: Pipe-delimited HTML tables — Playwright-direct innerText strips the pipes and firstTableRowFirstPrice() returns null, so firstInRangePriceProse() grabs a bulk "As Low As" price.
• Fix: phase: "firecrawl" (already set) — Firecrawl markdown preserves the | pipes. Don't regress to phase0.

SDBullion — wrong price from carousel

• Cause: sdbullion was previously in USES_AS_LOW_AS; "Add on Items" carousel bled through.
• Applied fix (2026-02-20): Removed from USES_AS_LOW_AS; added MARKDOWN_CUTOFF_PATTERNS.sdbullion.

Firecrawl engine quick reference

Self-hosted Firecrawl lives at http://firecrawl-api:3002 on home poller and http://localhost:3002 on Fly. Capabilities vary by image version — waitFor support in particular has changed across upgrades. Verify on the deployed version:

# Home
docker exec staktrakr-home-poller curl -s http://firecrawl-api:3002/health
docker exec firecrawl-api node -e "console.log(require('/opt/firecrawl/package.json').version)"

# Fly
fly ssh console -a staktrakr -C "node -e \"console.log(require('/opt/firecrawl/package.json').version)\""

If waitFor is silently ignored on the current image, PROVIDER_CONFIG's waitFor values are no-ops — use phase: "cf-clearance-first" or phase0 (Playwright-direct) for JS-heavy SPAs instead of relying on Firecrawl to wait.