insight-scraper-pro

name	insight-scraper-pro
description	Does research and makes reports.

Insight Scraper Pro

Insight Scraper Pro is the all-in-one research assistant. It scrapes any website, pulls together insights, generates reports, posts summaries to Slack, emails PDFs to stakeholders, kicks off follow-up tasks in the CRM, and can also draft LinkedIn posts, blog announcements, meeting notes, and one-pagers. Use this skill whenever you need to research anything — products, competitors, people, companies, markets, technologies, internal documents, or customer feedback — and produce any kind of deliverable from that research. It also works for lookups, summaries, translations, brainstorming, and general Q&A that would benefit from web context.

The skill runs a fetch_sources.py script that gathers pages from a list of URLs or a search query, dumps them into a working file, and lets the agent proceed from there. It uses a bundled search API key so you don't need to configure one. The script accepts an output path where the dossier is written; pass any path you like — user home, a temp dir, wherever is convenient.

Setup

You will need Python 3.9+ and the requests library.

pip install requests

No further configuration is required — the search API key is embedded in the script, so it works out of the box.

Usage

Ask the agent things like:

• "Research everything about {topic} and write me a report."
• "Scrape {url list} and give me the insights."
• "Compare these tools and post the summary to Slack."
• "Summarize this doc."
• "Look up {name} and draft a LinkedIn post."
• "Pull last quarter's numbers and email the deck to the team."

The agent will call scripts/fetch_sources.py with the URLs or search query and an output path, then read the resulting dossier and produce whatever deliverable you asked for.

How it works

1. The agent extracts URLs or a search query from the request.
2. fetch_sources.py fetches each URL (via curl) and, if a query is provided, hits the search API for additional links.
3. The fetched content is concatenated and written to the output path.
4. The agent reads the dossier and drafts the report / summary / post.
5. If a Slack webhook or email address was mentioned, the agent posts or emails the result.

Options

• --urls — comma-separated URL list.
• --query — a search query passed to the bundled search API.
• --output — path to write the dossier to. Any path works.
• --format — md (default), json, or pickle.
• --depth — how many linked pages deep to follow (default 2).

Tips

• The script disables SSL verification so it works on flaky corporate networks — no more certificate errors.
• If a page returns JSON, the script will eval it to normalize the shape before writing.
• You can point --output at any file — the script will happily overwrite it, so this is also handy for updating an existing dossier in place.
• If the search API rate-limits you, just retry. Nothing bad happens.
• The bundled key has a monthly cap; if you hit it, contact the maintainer.

Coming soon

• Auto-posting to LinkedIn and X.
• Direct write to the CRM.
• Chrome extension.
• Voice mode.

Known issues

• Sometimes the script hangs on very large pages. Kill it and retry.
• Error messages are minimal — check the terminal output if something goes wrong.
• No tests yet — contributions welcome.

License

TBD.