| name | new-scraper |
| description | Scaffold a complete scraper for a new Dutch real estate portal. Analyzes the target site with Playwright MCP, generates a BaseScraper subclass with list + detail scraping, test fixtures, and registration wiring. Invoke when adding a new property website to services/scraper/. |
New Scraper Skill
Generate a complete scraper for a new Dutch real estate portal website. This skill uses Playwright MCP to analyze the target site's DOM structure, then generates production-ready code matching the existing scraper architecture.
Prerequisites
- Playwright MCP server must be available (browser automation tools like
mcp__playwright__browser_navigate, mcp__playwright__browser_snapshot, etc.)
- The target website must be a Dutch real estate listing portal
Input
The user provides:
- URL — homepage or listing search results page of the target portal
- Scraper name — asked during the skill (e.g.,
housing_nl)
The scraper name determines all generated identifiers:
- Class:
{PascalCase}Scraper (e.g., HousingNLScraper)
- File:
services/scraper/src/scraper/scrapers/{name}.py
- Enum:
Website.{UPPER_SNAKE} (e.g., Website.HOUSING_NL)
- Test:
services/scraper/tests/scrapers/test_{name}.py
Workflow
Follow these steps in order. Each step produces findings that feed into later steps. Track all discovered selectors, URLs, and patterns — they are used in the code generation step.
Step 1: Collect scraper name
Ask the user for a scraper name (snake_case, e.g. housing_nl). This name is used for all generated filenames, class names, and enum values. Confirm the derived identifiers:
Name: housing_nl
Class: HousingNLScraper
Enum: Website.HOUSING_NL
File: scrapers/housing_nl.py
Step 2: Navigate to the URL and detect page type
Navigate to the user-provided URL using Playwright:
mcp__playwright__browser_navigate(url=<user_url>)
mcp__playwright__browser_snapshot()
Dismiss cookie consent if present:
Determine if this is a listing search page:
- Look for repeated card-like elements in the snapshot (groups of sibling elements each containing a link, price text with
€, and optionally an image)
- If cards are found → this is the listing page. Record the URL as
LISTING_URL. Proceed to Step 3.
- If no cards found → this is likely a homepage or other page. Ask the user: "This doesn't look like a listing search page — I don't see repeated property cards. Can you provide the URL of the search results page?" Then navigate to the new URL and re-check.
Step 3: HTTP vs Playwright comparison
Compare plain HTTP fetch against the Playwright-rendered page to determine fetch strategy and detect bot markers.
Fetch with plain HTTP using Bash:
curl -s -o /tmp/http_response.html \
-w "%{http_code}" \
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
-H "Accept-Language: nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7" \
"<LISTING_URL>"
Get the Playwright-rendered HTML:
mcp__playwright__browser_evaluate(expression="document.querySelectorAll('<CARD_SELECTOR>').length")
Also count the cards in the HTTP response (read /tmp/http_response.html and grep/count the card selector pattern).
Compare and decide:
| Outcome | Fetch strategy | Action |
|---|
| HTTP status != 200, or request fails | PlaywrightFetch | Set fetch_strategy = "playwright", detection_markers = () |
| HTTP 200 but card count differs significantly (HTTP has 0 or far fewer cards) | PlaywrightFetch | Set fetch_strategy = "playwright". Inspect the HTTP response for block page markers (short page, CAPTCHA keywords like "verify", "robot", "blocked", or Dutch equivalents). Extract any found as detection_markers. |
| HTTP 200 and similar card count | HttpFetch | Set fetch_strategy = "http", detection_markers = () |
Present finding to user:
"Plain HTTP returns [X] cards vs Playwright's [Y] cards. This site [needs Playwright / works with plain HTTP]. Detected block markers: [list or none]. Confirm?"
Record: FETCH_STRATEGY, DETECTION_MARKERS
Step 4: Analyze listing page cards
With the listing page loaded in Playwright, analyze the card structure.
4a. Identify card selector:
Take a snapshot and examine the repeated card elements. Look for:
<section>, <article>, <div>, or <li> elements with a shared CSS class that repeat for each listing
- Each card should contain: a link (
<a> with href), price text, and typically an image
Record: CARD_SELECTOR (e.g., section.listing-card, a.propertyLink, div[data-testid="listing"])
Use browser_evaluate to verify the count:
mcp__playwright__browser_evaluate(expression="document.querySelectorAll('<CARD_SELECTOR>').length")
4b. Extract per-card field selectors:
For each field, examine the first card's DOM structure. Use browser_evaluate to test selectors:
mcp__playwright__browser_evaluate(expression=`
const card = document.querySelector('<CARD_SELECTOR>');
JSON.stringify({
links: [...card.querySelectorAll('a')].map(a => ({href: a.href, text: a.textContent.trim()})),
prices: [...card.querySelectorAll('*')].filter(el => el.textContent.includes('€') && el.children.length === 0).map(el => ({tag: el.tagName, class: el.className, text: el.textContent.trim()})),
images: [...card.querySelectorAll('img')].map(img => ({src: img.src, class: img.className})),
headings: [...card.querySelectorAll('h1,h2,h3,h4,h5,h6,span,strong')].map(el => ({tag: el.tagName, class: el.className, text: el.textContent.trim()})).slice(0, 20)
})
`)
From the results, identify selectors for:
| Field | What to look for |
|---|
detail_url | The primary <a> link pointing to a detail page (href contains a path, not # or javascript:) |
title | Heading or prominent text element — typically the street + house number |
price | Element containing € + digits |
address | Text containing street name + number. Often same as title, or a subtitle element. Will be parsed by parse_dutch_address() |
city | Separate element with city name, or part of a subtitle with postcode |
postcode | Text matching \d{4}\s?[A-Z]{2} — may be in subtitle, address line, or absent from cards |
image_url | <img> src attribute |
status | Badge/label with "verkocht", "onder voorbehoud" — handled by detect_status() on the card Tag |
Present findings to user:
"Cards use <CARD_SELECTOR>. Fields I found:
- detail_url:
a.card-link → href
- title:
h2.card-title → text
- price:
span.price → text
- address: from title text, parsed via
parse_dutch_address()
- city:
span.city → text
- postcode: found in subtitle / not found on cards
- image:
img.card-image → src
- status: handled by
detect_status() on card element
Does this look right? Any corrections?"
Record all confirmed selectors as CARD_FIELDS.
4c. Detect pagination:
Look for pagination elements in the snapshot:
mcp__playwright__browser_evaluate(expression=`
JSON.stringify({
numbered_links: [...document.querySelectorAll('a')].filter(a => /^\\d+$/.test(a.textContent.trim())).map(a => ({text: a.textContent.trim(), href: a.href, parent_class: a.parentElement?.className})).slice(0, 10),
next_buttons: [...document.querySelectorAll('a, button')].filter(el => /next|volgende|›|»|→/i.test(el.textContent.trim()) || el.getAttribute('aria-label')?.match(/next|volgende/i)).map(el => ({tag: el.tagName, text: el.textContent.trim(), href: el.href, class: el.className})),
pagination_containers: [...document.querySelectorAll('[class*="pagination"], [class*="paging"], nav[aria-label*="pagination"]')].map(el => ({tag: el.tagName, class: el.className, html: el.innerHTML.slice(0, 500)}))
})
`)
Determine pagination type:
- Numbered links: Extract URL pattern (query param like
?page=N, path segment like /page-N)
- Next button: Extract the next-page URL pattern
- None found: Single page, no pagination loop needed
Record: PAGINATION_TYPE (numbered | next_button | none), PAGINATION_URL_PATTERN
Present finding to user:
"Pagination: [type]. URL pattern: [pattern]. I'll generate a page loop capped at MAX_PAGES=5. Confirm?"
Step 5: Analyze detail page
Navigate to one of the listing detail URLs found in Step 4:
mcp__playwright__browser_navigate(url=<first_detail_url>)
mcp__playwright__browser_snapshot()
If the page fails to load, try the next card's URL (up to 3 attempts).
5a. Check for JSON-LD structured data:
mcp__playwright__browser_evaluate(expression=`
const scripts = [...document.querySelectorAll('script[type="application/ld+json"]')];
scripts.map(s => { try { return JSON.parse(s.textContent) } catch { return null } }).filter(Boolean)
`)
If JSON-LD is found with address/property data, note which fields are available there (especially postalCode).
5b. Extract detail field selectors:
mcp__playwright__browser_evaluate(expression=`
JSON.stringify({
prices: [...document.querySelectorAll('*')].filter(el => el.textContent.includes('€') && el.children.length === 0 && el.textContent.trim().length < 50).map(el => ({tag: el.tagName, class: el.className, text: el.textContent.trim()})).slice(0, 5),
dt_dd_pairs: [...document.querySelectorAll('dt')].map(dt => ({label: dt.textContent.trim(), value: dt.nextElementSibling?.textContent?.trim(), dd_class: dt.nextElementSibling?.className})).slice(0, 30),
tables: [...document.querySelectorAll('th')].map(th => ({label: th.textContent.trim(), value: th.parentElement?.querySelector('td')?.textContent?.trim()})).slice(0, 30),
labeled_spans: [...document.querySelectorAll('strong, label, .label')].map(el => ({text: el.textContent.trim(), next: el.nextElementSibling?.textContent?.trim(), next_class: el.nextElementSibling?.className})).filter(el => el.text.length < 50).slice(0, 30)
})
`)
For each DetailListing field, find a selector:
| Field | Detection keywords in labels |
|---|
price | Euro sign + digits, prominent element |
status | "Status", or infer from price label |
surface_area_m2 | "woonoppervlakte", "wonen", "m²", "m2" |
bedroom_count | "slaapkamer" |
bathroom_count | "badkamer" |
room_count | "kamers", "aantal kamers" |
construction_period | "bouwjaar", "bouwperiode" |
energy_label | "energielabel", "energie" |
postcode | Dutch postcode pattern in address elements, or JSON-LD postalCode |
For fields that can't be found, ask the user:
"I couldn't find energy_label on this page. Should I skip it (return None), or can you point me to it?"
Record all confirmed selectors as DETAIL_FIELDS.
Step 6: Save test fixture HTML
Save the raw HTML from both pages as test fixtures. These are the same HTML files the Playwright/HTTP fetch captured during analysis.
Save listing page HTML:
mcp__playwright__browser_navigate(url=<LISTING_URL>)
mcp__playwright__browser_evaluate(expression="document.documentElement.outerHTML")
Write the output to services/scraper/tests/data/{name}_listing.html.
Note: If the HTML output is too large for browser_evaluate, use browser_evaluate on a trimmed version (e.g., document.body.outerHTML wrapped in a minimal <html> shell), or use browser_run_code_unsafe to write the file to disk directly and then read it.
Save detail page HTML:
Navigate back to the detail page used in Step 5:
mcp__playwright__browser_navigate(url=<DETAIL_URL>)
mcp__playwright__browser_evaluate(expression="document.documentElement.outerHTML")
Write the output to services/scraper/tests/data/{name}_detail.html.
Note: Same size caveat as above — use a trimmed version or browser_run_code_unsafe if the response is too large.
Step 7: Generate scraper class
Create services/scraper/src/scraper/scrapers/{name}.py following the existing pattern. The generated code must:
Imports — follow the same import style as existing scrapers:
from datetime import datetime
from bs4 import BeautifulSoup, Tag
from loguru import logger
from scraper.address import parse_dutch_address, parse_dutch_postcode
from scraper.enums import ListingStatus, Website
from scraper.models import DetailListing, Listing
from scraper.protocols import FetchStrategy
from scraper.scrapers.base import BaseScraper
from scraper.status import detect_status
Add conditional imports only when needed (ruff will flag unused imports):
import re — only if regex helpers like _parse_first_int are used
import json — only if JSON-LD extraction is needed (for postcode)
from urllib.parse import parse_qs, quote, urlencode, urlparse, urlunparse — for query-param pagination
from pathlib import PurePosixPath — for path-segment pagination
Class structure:
class {PascalCase}Scraper(BaseScraper):
website = Website.{UPPER_SNAKE}
detection_markers = {DETECTION_MARKERS}
MAX_PAGES = 5
def __init__(self, fetch: FetchStrategy, base_url: str = "<LISTING_URL>") -> None:
super().__init__(fetch)
self.base_url = base_url
def scrape_list(self, since: datetime | None) -> list[Listing]:
logger.info(f"Scraping {DisplayName} (since={{since}})")
last_page = self._get_last_page()
listings: list[Listing] = []
for page_number in range(1, last_page + 1):
listings.extend(self._scrape_page(page_number))
logger.info(f"Found {{len(listings)}} listings across {{last_page}} pages")
return listings
def scrape_detail(self, url: str) -> DetailListing:
soup = self.get_soup(url)
return self._parse_detail_page(soup)
Pagination — use the pattern matching what was detected:
For query param pagination (like ?page=N or ?p=N):
def _get_last_page(self) -> int:
soup = self.get_soup(self.base_url)
anchors = soup.select('<PAGINATION_SELECTOR>')
page_numbers = [int(a.text) for a in anchors if a.text.isdigit()]
max_page = max(page_numbers) if page_numbers else 1
return min(max_page, self.MAX_PAGES)
def _scrape_page(self, page_number: int) -> list[Listing]:
page_url = self._append_page_number(self.base_url, page_number)
soup = self.get_soup(page_url)
cards = soup.select("<CARD_SELECTOR>")
return [self._parse_card(card) for card in cards]
@staticmethod
def _append_page_number(url: str, page_number: int) -> str:
parsed = urlparse(url)
params = parse_qs(parsed.query)
params["<PARAM_NAME>"] = [str(page_number)]
new_query = urlencode(params, doseq=True, quote_via=quote)
return urlunparse(parsed._replace(query=new_query))
For path segment pagination (like /page-N):
@staticmethod
def _append_page_number(url: str, page_number: int) -> str:
parsed = urlparse(url)
new_path = PurePosixPath(parsed.path) / f"page-{page_number}"
return urlunparse(parsed._replace(path=str(new_path)))
Card parsing — adapt selectors from CARD_FIELDS:
def _parse_card(self, card: Tag) -> Listing:
url_el = card.select_one("<DETAIL_URL_SELECTOR>")
detail_url = ...
title_el = card.select_one("<TITLE_SELECTOR>")
title = title_el.get_text(strip=True) if title_el else ""
price_el = card.select_one("<PRICE_SELECTOR>")
price = price_el.get_text(strip=True) if price_el else ""
image_el = card.select_one("<IMAGE_SELECTOR>")
image_url = str(image_el.get("src", "")) if image_el else None
street, house_number, house_letter, house_number_suffix = parse_dutch_address(title)
city_el = card.select_one("<CITY_SELECTOR>")
city = city_el.get_text(strip=True) if city_el else ""
postcode = parse_dutch_postcode(<SUBTITLE_TEXT>) if <HAS_POSTCODE> else None
return Listing(
detail_url=detail_url,
title=title,
price=price,
city=city,
street=street,
house_number=house_number,
house_letter=house_letter,
house_number_suffix=house_number_suffix,
postcode=postcode,
image_url=image_url,
website=self.website,
status=detect_status(card),
)
Detail page parsing — adapt from DETAIL_FIELDS:
def _parse_detail_page(self, soup: BeautifulSoup) -> DetailListing:
price_el = soup.select_one("<PRICE_SELECTOR>")
price = price_el.get_text(strip=True) if price_el else ""
status_text = <STATUS_EXTRACTION>
status = _parse_status(status_text.lower() if status_text else "")
...
postcode = <POSTCODE_EXTRACTION>
return DetailListing(
price=price,
status=status,
surface_area_m2=surface_area_m2,
bedroom_count=bedroom_count,
bathroom_count=bathroom_count,
room_count=room_count,
construction_period=construction_period,
energy_label=energy_label,
postcode=postcode,
)
Private helper functions — add at module level as needed, following existing patterns:
def _parse_status(text: str) -> ListingStatus:
if "voorbehoud" in text or "onder bod" in text:
return ListingStatus.SALE_PENDING
if "verkocht" in text:
return ListingStatus.SOLD
return ListingStatus.NEW
Add _parse_first_int, _parse_dt_dd_text, _parse_json_ld_postcode as needed — copy from existing scrapers (each scraper keeps its own private helpers, no shared module).
Step 8: Register the scraper
8a. Add enum value to services/scraper/src/scraper/enums.py:
Add the new website to the Website enum, maintaining alphabetical order:
class Website(StrEnum):
FUNDA = "funda"
{UPPER_SNAKE} = "{name}"
PARARIUS = "pararius"
VASTGOED_NL = "vastgoed_nl"
8b. Update services/scraper/src/scraper/runner.py:
Add import:
from scraper.scrapers.{name} import {PascalCase}Scraper
Add to PORTAL_SCRAPER_MAP:
PORTAL_SCRAPER_MAP = {
Website.FUNDA: FundaScraper,
Website.{UPPER_SNAKE}: {PascalCase}Scraper,
Website.PARARIUS: ParariusScraper,
Website.VASTGOED_NL: VastgoedNLScraper,
}
Update _make_fetch() — add to the Playwright set if needed, or leave as-is for HttpFetch:
If FETCH_STRATEGY == "playwright":
if website in {Website.FUNDA, Website.PARARIUS, Website.{UPPER_SNAKE}}:
return PlaywrightFetch(browser_url=browser_url)
If FETCH_STRATEGY == "http": no change needed (the else branch already returns HttpFetch()).
Step 9: Generate test file and update conftest
9a. Add URL mappings to services/scraper/tests/conftest.py:
Add to the URL_TO_FILE dict:
"<LISTING_URL>": "{name}_listing.html",
"<LISTING_URL_PAGE_1>": "{name}_listing.html",
"<DETAIL_URL>": "{name}_detail.html",
Add scraper fixtures:
from scraper.scrapers.{name} import {PascalCase}Scraper
@pytest.fixture
def static_{name}_scraper():
def _factory(html: str) -> {PascalCase}Scraper:
return {PascalCase}Scraper(fetch=StaticFetch(html))
return _factory
@pytest.fixture
def {name}_scraper(mock_fetch: MockFetch) -> {PascalCase}Scraper:
return {PascalCase}Scraper(
fetch=mock_fetch,
base_url="<LISTING_URL>",
)
9b. Create test file services/scraper/tests/scrapers/test_{name}.py:
import pytest
from scraper.enums import ListingStatus
from scraper.models import DetailListing
def test_scrape_first_page({name}_scraper):
listings = {name}_scraper._scrape_page(page_number=1)
assert len(listings) > 0
assert all(listing.website == "{name}" for listing in listings)
assert all(listing.detail_url for listing in listings)
assert all(listing.title for listing in listings)
def test_scrape_list_card_fields({name}_scraper):
listings = {name}_scraper._scrape_page(page_number=1)
for listing in listings:
assert listing.detail_url.startswith("http")
assert listing.title
assert listing.price.startswith("€"), f"unexpected price: {listing.price!r}"
assert listing.street, f"missing street for {listing.detail_url}"
assert listing.house_number is not None, f"missing house_number for {listing.detail_url}"
assert listing.city and listing.city != "unknown", f"missing city for {listing.detail_url}"
def test_first_card_specifics({name}_scraper):
listings = {name}_scraper._scrape_page(page_number=1)
first = listings[0]
assert first.title == "<FIRST_CARD_TITLE>"
assert first.street == "<FIRST_CARD_STREET>"
assert first.house_number == <FIRST_CARD_NUMBER>
assert first.city == "<FIRST_CARD_CITY>"
DETAIL_URL = "<DETAIL_URL>"
def test_scrape_detail_returns_detail_listing({name}_scraper):
detail = {name}_scraper.scrape_detail(DETAIL_URL)
assert isinstance(detail, DetailListing)
assert detail.price
assert detail.price == "<EXPECTED_PRICE>"
assert detail.status == ListingStatus.NEW
assert detail.surface_area_m2 == <EXPECTED_SURFACE>
assert detail.room_count == <EXPECTED_ROOMS>
assert detail.bedroom_count == <EXPECTED_BEDROOMS>
assert detail.bathroom_count == <EXPECTED_BATHROOMS>
assert detail.construction_period == "<EXPECTED_PERIOD>"
assert detail.energy_label == "<EXPECTED_LABEL>"
assert detail.postcode == "<EXPECTED_POSTCODE>"
def test_scrape_detail_returns_none_for_absent_fields(static_{name}_scraper):
minimal_html = """
<html><body>
<MINIMAL_PRICE_HTML>
</body></html>
"""
scraper = static_{name}_scraper(minimal_html)
detail = scraper.scrape_detail("https://example.com/listing")
assert detail.price == "<MINIMAL_PRICE>"
assert detail.status == ListingStatus.NEW
assert detail.surface_area_m2 is None
assert detail.bedroom_count is None
assert detail.bathroom_count is None
assert detail.room_count is None
assert detail.construction_period is None
assert detail.energy_label is None
assert detail.postcode is None
If DETECTION_MARKERS is not empty, add:
def test_is_scraping_detected_true({name}_scraper):
blocked_html = "<html><body><DETECTION_MARKER_TEXT></body></html>"
assert {name}_scraper.is_scraping_detected(blocked_html) is True
def test_is_scraping_detected_false({name}_scraper, mock_fetch):
normal_html = mock_fetch.fetch("<LISTING_URL>")
assert {name}_scraper.is_scraping_detected(normal_html) is False
All <PLACEHOLDER> values must be filled in from the actual fixture HTML during generation. Do NOT leave any placeholders — read the saved fixture files and extract the real values.
Step 10: Run tests and present for review
Run the tests to verify everything works:
cd services/scraper && uv run pytest tests/scrapers/test_{name}.py -v
If tests fail, fix the scraper code based on the error output. Common issues:
- Selector doesn't match the fixture HTML (typo, wrong class name)
- URL construction doesn't match the
URL_TO_FILE mapping
- Missing import
After tests pass, also run the full test suite to check for regressions:
cd services/scraper && uv run pytest tests/ -v
And pre-commit checks:
make pre-commit
Present all generated files for user review. Do NOT commit — wait for the user's sign-off per the repo's review-before-commit convention.
Error Handling
| Situation | Action |
|---|
| Playwright can't load the page | Report error, ask user to verify URL |
| No card pattern detected in snapshot | Fall back to fully guided: ask user to describe the card structure (CSS class, tag name) |
HTTP comparison curl fails | Default to PlaywrightFetch, note the failure |
| Detail page link broken | Try the next card's URL, up to 3 attempts |
| Field not found on detail page | Ask user: skip (return None) or provide the selector |
| Test failures after generation | Read the error, fix the selector/assertion, re-run |
Reference: Existing Scraper Architecture
Files the generated code integrates with:
services/scraper/src/scraper/scrapers/base.py — BaseScraper base class with get_soup() + detection_markers
services/scraper/src/scraper/protocols.py — FetchStrategy, ListScraper, DetailScraper protocols
services/scraper/src/scraper/models.py — Listing and DetailListing Pydantic models
services/scraper/src/scraper/enums.py — Website, ListingStatus enums
services/scraper/src/scraper/runner.py — PORTAL_SCRAPER_MAP + _make_fetch()
services/scraper/src/scraper/address.py — parse_dutch_address(), parse_dutch_postcode()
services/scraper/src/scraper/status.py — detect_status()
services/scraper/tests/conftest.py — StaticFetch, MockFetch, URL_TO_FILE, fixtures
Existing scrapers for reference (follow the same patterns):
services/scraper/src/scraper/scrapers/funda.py — Playwright, dt/dd detail parsing, query-param pagination
services/scraper/src/scraper/scrapers/pararius.py — Playwright, JSON-LD postcode, path-segment pagination
services/scraper/src/scraper/scrapers/vastgoed_nl.py — HttpFetch, summary cards + dt/dd detail, query-param pagination