CrawlSage

Handle JavaScript-heavy / "dynamic" pages in CrawlSage WITHOUT a browser — by extracting the data SSR/hydration frameworks embed in the page (Next.js __NEXT_DATA__, Nuxt __NUXT__, JSON-LD, inline state) or replaying the JSON API behind it. Use when a page looks empty in Http.fetch, renders client-side, loads via XHR, or is a SPA. Covers the Extract module and the rendering ladder.

Handle login, cookies and sessions in CrawlSage. Use when a crawl needs authentication — form login, keeping a session across requests, persisting a cookie jar, CSRF tokens, or bearer/API tokens. Covers the Session module (CookieContainer-backed client + form-POST login).

Make a CrawlSage crawl robust and polite — retries, back-off, throttling/rate limits, timeouts, robots.txt, proxy and User-Agent rotation. Use when a crawl is getting blocked/rate-limited, failing intermittently, hammering a site too fast, or needs to respect robots.txt. Covers Polly-based resilience composed around Http.fetch.

Parse and extract data from HTML in CrawlSage using AngleSharp with CSS selectors. Use when the user wants to scrape fields/lists/tables from a page, select elements, read text or attributes, or build/extend the Html parsing module. Covers the F#-idiomatic selector DSL.

Export scraped data from CrawlSage to JSON, JSON Lines, CSV or a database, and into Deedle data frames for analysis. Use when the user wants to save/persist/write scraped results, produce a CSV/JSON file, wire a Spider pipeline to disk, or post-process data. Covers the Export module and the Sink seam.

Scaffold a new CrawlSage crawler. Use when the user wants to start a new spider/crawler/scraper, add a runnable example under samples/, or says "create a crawler for <site>". Wires an F# console project to CrawlSage's Spider engine (fetch → parse → pipeline).