| name | Structured Extraction |
| description | Use when the user wants tables, lists, or JSON pulled from fetched pages, feeds, or APIs—web_fetch then parse cleanly. |
| version | 1.0.0 |
| category | bundled |
| primary-tools | ["web_fetch","run_python","write_file"] |
| tags | ["extraction","scraping","parsing","etl","web_fetch","data"] |
| triggers | ["scrape","extract table","parse html","structured data","csv from page","json from feed","normalize results","dedupe rows","list from page","web_fetch","write_file"] |
Tool contract (read first)
| Step | Tool |
|---|
| Fetch source | web_fetch (or URLs from open-web-research) |
| Parse on Nodebox | run_python + beautifulsoup4/lxml (auto-loaded) — see browser-runtime-map |
| Persist rows | write_file under work/<slug>/ |
| Deliver to user | artifact_present — artifact-delivery |
| Scholarly APIs | defer to research-pack |
Non-negotiable: Do not store extracted rows in memory_save — file + present. Prefer feeds/JSON-LD over regex on minified JS.
When to Use
- Converting
web_fetch or web_search output into typed rows (creators, products, papers, prices, listings).
- Deduping or normalizing entity lists produced by
open-web-research or research-pack.
- User asks for CSV, JSON, or table from a URL — not freeform summary.
Relation to other skills
open-web-research finds the URLs and confirms regional/topic fit; this skill shapes the rows.research-pack owns arXiv / Semantic Scholar API parsing — defer there for scholarly fields.- Tool-choice constraints live in
browser-runtime-map (no npx/curl on Nodebox).
Procedure
- Inspect raw
web_fetch.text first — note whether HTML, RSS/Atom, or JSON. Do not parse blind.
- Prefer feeds/APIs over HTML: RSS/Atom, sitemap.xml, JSON endpoints, oEmbed. Cheaper, stabler, less noise.
- Parse narrowly. On Nodebox use
run_python with beautifulsoup4 or lxml (Pyodide auto-loaded) for HTML; for JSON/RSS/Atom parse with stdlib json/xml.etree. Target regex only on canonical tags or JSON-LD <script type="application/ld+json">.
- Normalize before emitting: lowercase hostnames, strip tracking params, ISO-8601 dates, trimmed text, stable key order.
- Dedupe by canonical URL or id, not display title.
- Emit JSON array plus a short pipe-table summary for the user.
Output template
[
{ "id": "...", "title": "...", "url": "https://...", "published_at": "2026-01-12", "source": "..." }
]
Followed by:
| Title | Source | Date |
|-------|--------|------|
Pitfalls
- Regex against minified JS bundles — brittle. Prefer JSON-LD or feed.
- Trusting the first search hit without
web_fetch confirm (same verify rule as open-web-research).
- Silent row drops on parse error — log skipped count instead.
- Mixing units / timezones — normalize at parse time, not in the table.
Anti-patterns
- Copying raw HTML into chat for the user to clean.
- Chaining multiple shell pipes when one
node -e does the parse.
- Storing extracted rows in
memory_save — write to a file under work/ and use artifact-delivery.
