| name | anakin-web-data |
| description | Use when fetching content from the web with Anakin — reading a page, getting a site's URL structure, or bulk-collecting many pages. Covers scrape, map, and crawl, and routes to the right Anakin tool for a task. Triggers on scraping a URL, extracting page content as markdown, listing a site's pages, ingesting a site, building a corpus from a domain, or handling JS-heavy/SPA pages. |
Anakin: fetching web data
Pick the right tool first
Anakin has twenty-one tools across five groups. Choosing wrong is the most
common and most expensive mistake — check this table before calling anything.
| The task | Tool | Skill |
|---|
| Read one known URL | scrape | this one |
| Find what pages exist on a site | map | this one |
| Collect many pages from one site | crawl | this one |
| Find pages across the web for a query | search | anakin-research |
| Answer a question needing many sources | agentic_search | anakin-research |
| See what AI engines say about something | ai_visibility_search | anakin-research |
| Extract data from a specific popular site | wire_discover → wire_read_action | anakin-wire |
| Submit a form, add to cart, post content | wire_discover → wire_write_action | anakin-wire |
| Watch a page for changes over time | monitor_create | anakin-monitoring |
| Multi-step interaction no Wire action covers | browser_task | anakin-browser |
| Reach login-protected content | session_list → pass sessionId | anakin-browser |
Check Wire before reaching for scrape or crawl on a well-known site.
Amazon, Walmart, LinkedIn, Airbnb, Zillow and hundreds more have vetted,
pre-built extractors that return clean structured data in one call. Scraping
those sites by hand is slower, costs more, and gives you markdown you then have
to parse. See the anakin-wire skill.
Escalate in cost order. scrape → scrape with useBrowser → a Wire
action → browser_task. Each step is slower and more expensive than the last;
do not start at the end.
scrape — one URL to markdown
Default behavior returns clean markdown. That is the right default; do not add
options reflexively.
generateJson: true — also run AI extraction and return typed fields. Use for
product pages, listings, articles — anywhere you want structured values rather
than prose. Returns the JSON as the primary output.useBrowser: true — render with a stealth headless browser. Slower and more
expensive. Only for SPAs and JS-rendered content. Try the default first; if
the markdown comes back empty or is missing the content you can see in a
browser, retry with useBrowser: true.country — two-letter proxy egress code, defaults to "us". Set it when a
page is geo-restricted or price/inventory varies by market ("de", "in", …).forceFresh: true — skip the cache. Results cache for roughly 24h, which is
usually what you want. Only force fresh for genuinely live data (stock levels,
prices you are about to act on).sessionId / sessionName — a saved browser session for login-protected
pages. Pair with useBrowser: true.
map — discover a site's URLs
Returns internal links, external links, and counts. Use it to understand
structure before crawling, so the crawl is scoped instead of blind.
limit (default 100, max 1000) — total URLs returned.depth (default 2, max 5) — link-hops followed.limitPerLevel (default 100) — breadth cap per level.search — keyword filter on path/title. Cheap way to find the section you
care about without crawling everything.includeSubdomains, includeExternalLinks — both default false.
crawl — bulk markdown from a site
For catalog ingestion or building a RAG corpus. Returns an array of pages, each
with markdown and its own status — check per-page status, since a crawl can
partially succeed.
maxPages (default 10, max 500) — a hard cap, and the main cost lever.depth (default 1, max 5).includePatterns / excludePatterns — glob/regex URL filters. Use these.
An unscoped crawl burns credits on nav pages, tag archives, and pagination.
Working pattern
For anything beyond a single page, map first, then crawl the subset:
map the domain with a search filter to see what exists.- Derive
includePatterns from the URLs that matter.
crawl with those patterns and a deliberate maxPages.
This is consistently cheaper and cleaner than crawling from the homepage and
filtering afterward.
Cost and failure notes
useBrowser is the biggest cost multiplier — leave it off unless the page
needs it.- Every tool returns an error envelope rather than throwing. On failure, read
the message: it names the tool and the cause.
- API key comes from
ANAKIN_API_KEY. Get one at https://anakin.io/dashboard —
free tier is 300 credits, no card.
