web-scraping

name	web-scraping
description	Plan and implement web scraping workflows with robust target clarification and selector strategy. Use when the user asks to scrape websites, extract page data, crawl pages, parse HTML, or build scraping scripts.

Web Scraping

Instructions

Use this skill when the user asks to scrape, crawl, extract, parse, or collect data from web pages.

Clarify the Target

Before implementation, make sure the scraping target is understood:

1. Ask what site, page type, or URL pattern should be scraped.
2. Ask what exact fields should be extracted.
3. Ask whether pagination, infinite scroll, login, filters, or multiple page templates are involved.
4. Ask what output format is expected, such as JSON, CSV, database rows, or an API response.
5. Ask about volume, rate limits, scheduling, and whether the site permits scraping.

If uncertain about the structure of the web page, always ask for example(s). Ask additional questions to make sure that you understand the target that the user requests to scrape.

Selector Rules

Always use some reliable selectors - not text, not aria attributes but ids, classes, HTML structure, etc.

Prefer selectors in this order:

1. Stable id attributes.
2. Stable semantic or component classes.
3. Data attributes if present, such as data-*.
4. HTML structure anchored on stable containers.
5. Attribute selectors for durable URLs, item types, or schema metadata.

Avoid selectors based on:

• Visible text content.
• aria-* attributes.
• Generated or obfuscated classes unless no better option exists.
• Positional selectors that depend on fragile sibling order unless scoped inside a stable repeated item container.

Implementation Guidance

Inspect the page structure before writing extraction logic. Use structured parsers for static HTML and browser automation only when JavaScript rendering or interaction is required.

When navigation is required, such as opening a new page, following a link, advancing pagination, or waiting for a routed view, use retry logic with bounded attempts, short backoff, and clear failure reporting.

For repeated items, select a stable parent container first, then query fields relative to each item. Keep selectors centralized so they are easy to update when the page changes.

Validate extraction against at least one representative example page. If there are multiple templates, validate against one example per template.

Handle missing optional fields explicitly. Treat missing required fields as a signal that the selector or page structure may have changed.

Response Pattern

When proposing or implementing a scraper, explain:

• The assumptions about page structure.
• The selectors being used and why they are stable.
• How pagination or dynamic loading is handled.
• The retry strategy used for navigation steps.
• How the extracted data is validated.