resilient-extraction-and-parsing

name

resilient-extraction-and-parsing

description

Extract web data defensively — prefer structured data (JSON-LD / microdata / __NEXT_DATA__ / JSON API) over brittle CSS/XPath selectors, anchor selectors on stable attributes with fallbacks, and validate every record to a schema so breakage is DETECTED not silently wrong. Traverses the parse branch of the web-scraping decision tree. Reach for this when the user asks 'build a robust extractor', 'this scraper keeps breaking', 'how do I parse this reliably?', or 'why is my scraped data wrong?'. Used by scraper-implementation-engineer (primary) and extraction-architect.

Skill: resilient-extraction-and-parsing

Invoked by: scraper-implementation-engineer (primary — the parser + validation) and extraction-architect (to judge a target's parse-fragility).

When to invoke: "build a robust extractor for this"; "this scraper keeps breaking"; "how do I parse this reliably?"; "why is my scraped data silently wrong?"; any "make extraction robust" question.

Output: a defensive extraction — structured-data-first parsing, selectors anchored with fallbacks, and per-record schema validation so breakage surfaces as a detected error.

Procedure

1. Reach for structured data before the DOM. In priority order: (a) the JSON/XHR API the page calls; (b) JSON-LD (<script type="application/ld+json">) — often has the clean entity; (c) embedded app state (__NEXT_DATA__, __NUXT__, a window.__STATE__); (d) microdata / RDFa / Open Graph meta. These have a documented-ish shape and survive redesigns that shatter CSS selectors.
2. Fall to selectors only when nothing structured exists — and anchor them. Prefer stable hooks: id, data-* attributes, ARIA roles, semantic elements. Avoid layout-position selectors (div > div:nth-child(3)) and auto-generated class names (hashed CSS-module names) — they break on the next deploy.
3. Add fallbacks per field. For each field, try the best source, then a fallback (e.g. JSON-LD price → [itemprop=price] → a labeled selector). A field with one fragile path is a single point of failure.
4. Define the target schema first, and validate every record. Declare the shape (field names, types, required/optional, ranges/formats). Validate each extracted record against it. A record that fails validation is quarantined and alerted, never written silently — this is the difference between "detected breakage" and "a month of wrong data."
5. Normalize at extraction. Parse prices/dates/units into canonical types (a Decimal price with currency, an ISO timestamp), trim whitespace, resolve relative URLs, decode entities. Normalize once, at the boundary, so downstream is clean.
6. Handle the absent and the malformed explicitly. Missing element ≠ empty string ≠ zero. Decide per field whether missing is valid (optional) or a validation failure, and never let a parser exception silently drop a whole record without a log.
7. Make breakage observable. Track per-run: records extracted, validation-failure rate, and empty/near-empty results. A spike means the site changed — that's the signal to fix the selector, caught in hours instead of discovered as missing data weeks later.

Worked example

User: "My CSS-selector scraper for article metadata keeps breaking every time the site redesigns."

• Root cause: selectors anchored on layout/hashed classes — they shatter on redesign.
• Structured-data-first: the articles expose JSON-LD NewsArticle with headline, datePublished, author — extract from that instead of the DOM (retrieval-dated 2026-07; JSON-LD presence verified per-site at use).
• Fallback: for pages missing JSON-LD, fall to Open Graph meta (og:title, article:published_time), then a [itemprop] selector — three tiers, not one fragile path.
• Schema: { headline: str(required), published_at: iso-datetime(required), author: str(optional) }; validate each record; quarantine + alert on failure.
• Normalize: parse datePublished to a UTC ISO timestamp; trim the headline; resolve author to a canonical string.
• Observability: alert if validation-failure-rate > 5% or a run returns near-zero records — the redesign is now caught the day it ships, not a month later.

Guardrails

• Structured data before selectors — JSON endpoint / JSON-LD / embedded state / microdata first.
• Anchor on stable attributes — never layout position or hashed class names.
• Fallbacks per field — one fragile path is a single point of failure.
• Validate to a schema; quarantine failures — a failed field is detected breakage, not a silent value.
• Normalize at the boundary — canonical types once, so downstream is clean.
• Missing ≠ empty ≠ zero — decide per field; never drop a record silently on a parser exception.
• Alert on validation-failure spikes and empty runs — detect the site change, don't discover it.