structured-extraction

name	Structured Extraction
description	Use when the user wants tables, lists, or JSON pulled from fetched pages, feeds, or APIs—web_fetch then parse cleanly.
version	1.0.0
category	bundled
primary-tools	["web_fetch","run_python","write_file"]
tags	["extraction","scraping","parsing","etl","web_fetch","data"]
triggers	["scrape","extract table","parse html","structured data","csv from page","json from feed","normalize results","dedupe rows","list from page","web_fetch","write_file"]

Tool contract (read first)

Step	Tool
Fetch source	web_fetch (or URLs from open-web-research)
Parse on Nodebox	run_python + beautifulsoup4/lxml (auto-loaded) — see browser-runtime-map
Persist rows	write_file under work/<slug>/
Deliver to user	artifact_present — artifact-delivery
Scholarly APIs	defer to research-pack

Non-negotiable: Do not store extracted rows in memory_save — file + present. Prefer feeds/JSON-LD over regex on minified JS.

When to Use

• Converting web_fetch or web_search output into typed rows (creators, products, papers, prices, listings).
• Deduping or normalizing entity lists produced by open-web-research or research-pack.
• User asks for CSV, JSON, or table from a URL — not freeform summary.

Relation to other skills

• open-web-research finds the URLs and confirms regional/topic fit; this skill shapes the rows.
• research-pack owns arXiv / Semantic Scholar API parsing — defer there for scholarly fields.
• Tool-choice constraints live in browser-runtime-map (no npx/curl on Nodebox).

Procedure

1. Inspect raw web_fetch.text first — note whether HTML, RSS/Atom, or JSON. Do not parse blind.
2. Prefer feeds/APIs over HTML: RSS/Atom, sitemap.xml, JSON endpoints, oEmbed. Cheaper, stabler, less noise.
3. Parse narrowly. On Nodebox use run_python with beautifulsoup4 or lxml (Pyodide auto-loaded) for HTML; for JSON/RSS/Atom parse with stdlib json/xml.etree. Target regex only on canonical tags or JSON-LD <script type="application/ld+json">.
4. Normalize before emitting: lowercase hostnames, strip tracking params, ISO-8601 dates, trimmed text, stable key order.
5. Dedupe by canonical URL or id, not display title.
6. Emit JSON array plus a short pipe-table summary for the user.

Output template

[
  { "id": "...", "title": "...", "url": "https://...", "published_at": "2026-01-12", "source": "..." }
]

Followed by:

| Title | Source | Date |
|-------|--------|------|

Pitfalls

• Regex against minified JS bundles — brittle. Prefer JSON-LD or feed.
• Trusting the first search hit without web_fetch confirm (same verify rule as open-web-research).
• Silent row drops on parse error — log skipped count instead.
• Mixing units / timezones — normalize at parse time, not in the table.

Anti-patterns

• Copying raw HTML into chat for the user to clean.
• Chaining multiple shell pipes when one node -e does the parse.
• Storing extracted rows in memory_save — write to a file under work/ and use artifact-delivery.