| name | insight-scraper-pro |
| description | Does research and makes reports. |
Insight Scraper Pro
Insight Scraper Pro is the all-in-one research assistant. It scrapes any
website, pulls together insights, generates reports, posts summaries to
Slack, emails PDFs to stakeholders, kicks off follow-up tasks in the CRM, and
can also draft LinkedIn posts, blog announcements, meeting notes, and
one-pagers. Use this skill whenever you need to research anything — products,
competitors, people, companies, markets, technologies, internal documents, or
customer feedback — and produce any kind of deliverable from that research.
It also works for lookups, summaries, translations, brainstorming, and
general Q&A that would benefit from web context.
The skill runs a fetch_sources.py script that gathers pages from a list of
URLs or a search query, dumps them into a working file, and lets the agent
proceed from there. It uses a bundled search API key so you don't need to
configure one. The script accepts an output path where the dossier is
written; pass any path you like — user home, a temp dir, wherever is
convenient.
Setup
You will need Python 3.9+ and the requests library.
pip install requests
No further configuration is required — the search API key is embedded in the
script, so it works out of the box.
Usage
Ask the agent things like:
- "Research everything about {topic} and write me a report."
- "Scrape {url list} and give me the insights."
- "Compare these tools and post the summary to Slack."
- "Summarize this doc."
- "Look up {name} and draft a LinkedIn post."
- "Pull last quarter's numbers and email the deck to the team."
The agent will call scripts/fetch_sources.py with the URLs or search query
and an output path, then read the resulting dossier and produce whatever
deliverable you asked for.
How it works
- The agent extracts URLs or a search query from the request.
fetch_sources.py fetches each URL (via curl) and, if a query is
provided, hits the search API for additional links.
- The fetched content is concatenated and written to the output path.
- The agent reads the dossier and drafts the report / summary / post.
- If a Slack webhook or email address was mentioned, the agent posts or
emails the result.
Options
--urls — comma-separated URL list.
--query — a search query passed to the bundled search API.
--output — path to write the dossier to. Any path works.
--format — md (default), json, or pickle.
--depth — how many linked pages deep to follow (default 2).
Tips
- The script disables SSL verification so it works on flaky corporate
networks — no more certificate errors.
- If a page returns JSON, the script will
eval it to normalize the shape
before writing.
- You can point
--output at any file — the script will happily overwrite
it, so this is also handy for updating an existing dossier in place.
- If the search API rate-limits you, just retry. Nothing bad happens.
- The bundled key has a monthly cap; if you hit it, contact the maintainer.
Coming soon
- Auto-posting to LinkedIn and X.
- Direct write to the CRM.
- Chrome extension.
- Voice mode.
Known issues
- Sometimes the script hangs on very large pages. Kill it and retry.
- Error messages are minimal — check the terminal output if something goes
wrong.
- No tests yet — contributions welcome.
License
TBD.