scrape-define

name	scrape-define
description	Create a new extraction spec from a URL — explore a detail page, discover fields, quick schema approval
argument-hint	[url] [what to extract]
allowed-tools	Agent, Skill, Bash, Read, Write, AskUserQuestion

You are helping the user quickly define what to extract from a website. Download 1 detail page, discover fields, and iterate on the schema in the terminal until approved.

Read ${CLAUDE_SKILL_DIR}/../scrape/references/python-environments.md.

The output is a draft spec folder with an approved schema and values (no stored pages — those are Stage 2's job). It can be expanded with /scrape-spec (more pages, variant comparison, navigation).

Hard constraints — never violate these:

• You MUST NOT fetch, read, grep, or parse any HTML file yourself. Page download is handled by the /scrape-explore-site subagent; field discovery is handled by the /scrape-analyze-page subagent. The main agent only orchestrates and consumes their outputs.

• This includes raw.html, rendered.html, *.cleaned.html, and any other saved page HTML. You may check whether a file exists, but you MUST NOT open it, print it, grep it, parse it, or run extraction scripts against it from the main agent.

• You MUST invoke /scrape-analyze-page as a subagent before building any schema. Building a schema from raw HTML without first running that subagent is a critical error.

• While waiting for /scrape-analyze-page, do NOT read HTML files, run clean_html.py, extract_metadata.py, or any inline parser code, even if the agent response suggests continuing unrelated work. Wait for the subagent to complete, then read only its saved JSON output.

• A plateau in tool_calls_completed in a read_agent response does not mean the subagent is stuck — it is likely in a long model-inference pass. Keep polling until status: completed.

Parse intent

From $ARGUMENTS, determine:

• target_url: the starting URL
• data_type: what is being extracted, always singular — whether the user names it or you infer it from the URL/page. If the user says "books" use "book", "products" → "product", "articles" → "article". If inferring, use the singular item noun (e.g. "product" not "products", "book" not "books"). Singularize before using anywhere.
• field_hints: any specific fields the user mentioned (may be empty)
• site_name: a short identifier (e.g. "books-toscrape", "realestate-listings") — derive from the site and data type

Step 1: Set up folders

Ensure site_name is unique:

BASE="books-toscrape"  # your derived site_name
NAME="$BASE"
N=2
while [ -d ".scrape/$NAME" ]; do NAME="${BASE}-${N}"; N=$((N+1)); done
echo "$NAME"

Create the folder structure ({data_type} must be singular, e.g. book not books):

.scrape/{site_name}/
  {data_type}/
.scrape/.work/{site_name}/
  explore/
  analyze-page/

Step 2: Get a detail page

Ask the user how to obtain a detail page via AskUserQuestion:

• question: "How should I get a detail page to analyze?"
• header: "Detail page"
• options:
  • Provide a URL — "I'll paste a detail page URL (e.g. a product page)."
  • Explore the site — "Find one automatically from the target URL."

If the user picks "Provide a URL": follow up with a plain-text prompt — "Paste the detail page URL." — and wait for their next message. Then download that page using download.py:

uv run ${CLAUDE_SKILL_DIR}/../scrape-explore-site/scripts/download.py --skill scrape-define --log-file .scrape/.work/{site_name}/explore/download.log <<'EOF'
[{"url": "USER_URL", "output_dir": ".scrape/.work/{site_name}/explore/pages/detail-1", "page_type": "detail"}]
EOF

If the user picks "Explore the site": Use a subagent to run the /scrape-explore-site skill with minimal counts. The subagent prompt should be:

Run /scrape-explore-site {target_url} .scrape/.work/{site_name}/explore 1 0

This downloads the homepage + 1 detail page into .scrape/.work/{site_name}/explore/pages/.

If the site is blocked, suggest Zyte. Only invoke /scrape-zyte-login if the user agrees. After it returns, re-run the scrape-explore-site subagent above.

Step 3: Analyze the detail page

Use rendered.html for analysis. If it doesn't exist, fall back to raw.html. Choose between them with a file-existence check only; do not read, grep, print, clean, or parse either HTML file in the main agent.

Run a subagent with the analysis skill. The subagent prompt MUST start with /scrape-analyze-page, and the subagent must invoke that skill; it is not acceptable for the main agent to run clean_html.py, extract_metadata.py, sed, grep, cat, or inline Python/lxml parsing as a substitute.

Agent(description="analyze detail-1 rendered", prompt="/scrape-analyze-page Extract data from .scrape/.work/{site_name}/explore/pages/detail-1/rendered.html and save it into .scrape/.work/{site_name}/analyze-page/detail-1.rendered.json")

After starting the analysis subagent, wait for it to finish. If the expected JSON file is not present, retry the /scrape-analyze-page subagent or report a blocker. Do NOT create the analysis JSON yourself from HTML.

Read the analysis result from .scrape/.work/{site_name}/analyze-page/detail-1.rendered.json. Do not inspect the downloaded page HTML or any cleaned HTML after the analysis subagent returns; the JSON result is the only allowed field-discovery input for the main agent.

Step 4: Build schema

From the analysis result, build a JSON Schema:

• Collect all field names and their types
• Map types: str → string, float → number, int → integer, list → array, dict → object
• Add description for each field (infer from context)
• Mark fields matching field_hints as "source": "requested", others as "source": "discovered"

Step 5: Quick schema check

Present the schema with values in the terminal. Group by requested/discovered:

Found {N} fields on {detail_page_url}:

Requested:
  title (string): "A Light in the Attic"
  price (string): "£51.77"

Discovered:
  rating (integer): 3
  category (string): "Poetry"
  description (string): "It's hard to imagine a world without..." (2340 chars)
  upc (string): "a897fe39b1053632"
  image_url (string): "https://books.toscrape.com/media/cache/fe/72/..."

Wait for user response. The user can:

• Approve: "looks good", "ok", "approve", etc. → go to step 6
• Drop fields: "drop description, image_url" → remove from schema, show updated
• Keep discovered fields: "keep rating" → change source to "requested"
• Rename: "rename upc to product_code" → rename in schema
• Edit descriptions: "price should be without currency symbol" → update description and value
• Other instructions: apply and show updated schema

Loop until the user approves. Re-display the schema after each change.

Step 6: Save spec

Data type spec

Write .scrape/{site_name}/{data_type}/spec.json. The output MUST be valid JSON Schema (draft/2020-12) — use exactly the structure shown below. Do NOT invent a custom format (e.g. a plain array of field objects). Include examples in each schema field — the user-approved value (with any corrections). This tells Stage 2 the expected format. Include examples for all fields. Truncate values longer than 200 chars with "..." — keep enough to show the format.

{
  "url": "https://books.toscrape.com",
  "data_type": "product",
  "html_variant": "rendered",
  "schema": {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
      "title": {
        "type": "string",
        "description": "Book title as displayed on the detail page",
        "source": "requested",
        "examples": ["A Light in the Attic"]
      },
      "price": {
        "type": "string",
        "description": "Book price without currency symbol",
        "source": "requested",
        "examples": ["51.77"]
      }
    }
  }
}

Site-level spec

Write .scrape/{site_name}/spec.json:

{
  "url": "https://books.toscrape.com",
  "data_types": ["{data_type}"]
}

Note: data_types only lists the primary data type. /scrape-spec adds "navigation" later.

Done

Spec draft saved to .scrape/{site_name}/:
  {data_type}: {N} fields

Run /scrape-spec .scrape/{site_name} to explore more pages and validate.