zillow

name

zillow

description

Proven scraping playbook for zillow.com /homes/for_sale/ search pages. CloudFront + PerimeterX (HumanSecurity) gates raw curl with HTTP 403 `x-px-blocked: 1`. A puppeteer.connect() to the user's real Chrome via the real Chrome debug port walks through the challenge — but only if you (a) clear poisoned cookies first and (b) warm up via Google referer. All listings ship in `__NEXT_DATA__`. Activate for any zillow.com search/listing scrape.

Zillow (zillow.com)

Server-rendered Next.js search pages behind CloudFront + PerimeterX. Once a session is flagged ("Access to this page has been denied"), every subsequent navigation in that browser context returns the deny page until cookies are cleared. The fix is mechanical: clear cookies, hit Google first, then go to the search URL. After that, all 41 results for a city search are sitting in __NEXT_DATA__ as a single JSON blob — no API replay, no scrolling tricks needed.

Detection

Signal	Value
CDN	CloudFront (server: CloudFront, via: 1.1 …cloudfront.net)
Anti-bot	PerimeterX / HumanSecurity (x-px-blocked: 1 on block, _px* cookies on allow)
Framework	Next.js (__NEXT_DATA__ JSON blob)
Auth	None for search results
Robots	Disallows /search-results, but /homes/ listing pages are public

Bare curl/fetch returns HTTP 403 with a 5.8KB CAPTCHA HTML. Even a real Chrome session gets stuck on "Access to this page has been denied" once a previous tab in that profile got flagged — the deny verdict is sticky in cookies.

Architecture

• /homes/for_sale/<city>-<state>/ is server-rendered. The full first page of ~41 listings is embedded in <script id="__NEXT_DATA__" type="application/json"> at props.pageProps.searchPageState.cat1.searchResults.listResults (or any nested listResults array — walk the tree).
• The zillow.com homepage and search pages share the same PerimeterX cookie (_pxhd, _px3). A clean session fetched via Google → search URL gets a clean PX challenge that auto-passes.
• Pagination uses ?searchQueryState=<urlencoded JSON> with a pagination.currentPage field. Cursor-style; not needed for a scout run.

Strategy used

• Phase 0 (curl): 403, x-px-blocked: 1. Skip.
• Phase 1 (framework recon): Confirmed Next.js, but couldn't fetch HTML to parse. Skip.
• Phase 2 (browser via the real Chrome debug port): Works after cookie clear + Google warmup. This is the canonical path.

Stealth config that works

import puppeteer from "puppeteer-core";

const browser = await puppeteer.connect({ browserURL: "http://127.0.0.1:9222" });
const page = await browser.newPage();
await page.setViewport({ width: 1400, height: 900 });

// CRITICAL: clear poisoned PX cookies before navigation.
// Without this, every goto() returns "Access to this page has been denied"
// if any previous tab in the profile triggered the PX challenge.
const cdp = await page.target().createCDPSession();
await cdp.send("Network.clearBrowserCookies");

// Warm up via Google so the referer looks organic
await page.goto("https://www.google.com/search?q=zillow+san+francisco+homes+for+sale",
  { waitUntil: "domcontentloaded", timeout: 60000 });
await new Promise(r => setTimeout(r, 1500));

// Now hit the search URL
await page.goto("https://www.zillow.com/homes/for_sale/San-Francisco-CA/",
  { waitUntil: "domcontentloaded", timeout: 60000 });
await new Promise(r => setTimeout(r, 5000));

// Sanity check — title should be the page title, not "Access to this page has been denied"
const t = await page.title();
if (/denied/i.test(t)) throw new Error("PX blocked — clear cookies and retry");

No stealth plugin, no proxy, no captcha solver. Real Chrome's TLS+history is enough.

Extraction

All 41 cards live in __NEXT_DATA__. Walk the JSON tree for the first array named listResults:

const listings = await page.evaluate(() => {
  const out = [];
  const nd = document.getElementById("__NEXT_DATA__");
  if (!nd) return out;
  const j = JSON.parse(nd.textContent);
  const stack = [j];
  while (stack.length) {
    const cur = stack.pop();
    if (!cur || typeof cur !== "object") continue;
    if (Array.isArray(cur.listResults)) {
      for (const r of cur.listResults) {
        out.push({
          zpid: r.zpid,
          address: r.address,
          price: r.price,
          unformattedPrice: r.unformattedPrice,
          beds: r.beds,
          baths: r.baths,
          area: r.area,
          latLong: r.latLong,
          statusType: r.statusType,
          detailUrl: r.detailUrl?.startsWith("http")
            ? r.detailUrl
            : "https://www.zillow.com" + (r.detailUrl || ""),
          imgSrc: r.imgSrc,
          brokerName: r.brokerName,
          hdpData: r.hdpData?.homeInfo ? {
            homeType: r.hdpData.homeInfo.homeType,
            homeStatus: r.hdpData.homeInfo.homeStatus,
            zestimate: r.hdpData.homeInfo.zestimate,
            daysOnZillow: r.hdpData.homeInfo.daysOnZillow,
            livingArea: r.hdpData.homeInfo.livingArea,
            lotAreaValue: r.hdpData.homeInfo.lotAreaValue,
          } : null,
        });
      }
      return out;
    }
    for (const k in cur) stack.push(cur[k]);
  }
  return out;
});

DOM fallback (if __NEXT_DATA__ ever moves): article[data-test="property-card"] with sub-selectors [data-test="property-card-addr"] and [data-test="property-card-price"].

Anti-blocking summary

Layer	Needed?	Notes
1 — Headers/UA	No	Real Chrome supplies them
2 — Stealth plugin	No	Real Chrome IS the user's Chrome
3 — Headed mode	Yes	Via the real Chrome debug port
4 — Real Chrome profile	Yes — required	Disposable Chromium gets PX-blocked
5 — Cookie hygiene	Yes — required	Must clear cookies before each run; PX deny verdict is sticky
6 — Referer warmup	Yes — required	Direct goto() to search URL fails; Google → search works
7 — CAPTCHA solver	No	Cookie clear + warmup avoids the challenge entirely
8 — Residential proxy	No	Home IP works
9 — Auth	No	Public listings

Data shape

{
  "zpid": "69817587",
  "address": "1440 Broadway APT 307, San Francisco, CA 94109",
  "price": "$799,999",
  "unformattedPrice": 799999,
  "beds": 1,
  "baths": 1,
  "area": 818,
  "latLong": { "latitude": 37.796387, "longitude": -122.42072 },
  "statusType": "FOR_SALE",
  "detailUrl": "https://www.zillow.com/homedetails/1440-Broadway-APT-307-San-Francisco-CA-94109/69817587_zpid/",
  "imgSrc": "https://photos.zillowstatic.com/fp/...-p_e.jpg",
  "hdpData": {
    "homeType": "CONDO",
    "homeStatus": "FOR_SALE",
    "zestimate": 795700,
    "daysOnZillow": 12,
    "livingArea": 818
  }
}

A San Francisco search returns 41 results on the first page.

Pagination / crawl architecture

• Default page size is ~41 cards per response.
• For more, build a searchQueryState JSON like:

  {"pagination":{"currentPage":2},"mapBounds":{...},"filterState":{...}}
  

  URL-encode and append as ?searchQueryState=…. Cap is page 20 (~800 results) per region.
• Better path for big crawls: split by zip/neighborhood URL (/<neighborhood>/) — Zillow caps each polygon at ~800 results, so splitting geographically beats deep pagination.
• Also available: undocumented POST https://www.zillow.com/async-create-search-page-state accepting searchQueryState + wants + requestId — same cookies, returns JSON directly. Worth replaying once you have a warm session.

Gotchas & lessons

1. The PX deny verdict is sticky in cookies. Once any tab hits "Access to this page has been denied", every subsequent zillow.com navigation in that profile returns the same page until you Network.clearBrowserCookies. This is the #1 thing that breaks reruns. Always clear cookies at the top of the script.
2. Direct goto(searchUrl) from a clean session sometimes still trips PX. Hitting Google first (any zillow-related query) and then the search URL is reliable. The 1.5s pause after Google matters.
3. Don't use puppeteer.launch() (disposable Chromium). PX scores it instantly even with the stealth plugin. Real Chrome via 127.0.0.1:9222 is the only working browser path observed.
4. waitUntil: "networkidle2" hangs on Zillow because of long-poll websockets. Use "domcontentloaded" + a fixed sleep.
5. No need to scroll to load cards — the entire listResults array is in __NEXT_DATA__ at page load. Scrolling only matters if you want the lazy-loaded image srcs, which imgSrc in the JSON already gives you.
6. detailUrl is sometimes relative (/homedetails/…) and sometimes absolute. Normalize both ways.
7. __NEXT_DATA__ schema is deeply nested. Walk for any array named listResults instead of hardcoding the path — Zillow has shuffled the path inside cat1.searchResults.* multiple times.
8. disconnect(), never close() the browser — close() would kill the user's real Chrome.