technews-webscrape
Configure and use reputable web scraping for TK TechNews. Use when an agent needs Firecrawl MCP, Firecrawl-backed scraping, dynamic page extraction, web source troubleshooting, or guidance on when to use RSS, local scraping, Firecrawl, or YouTube transcripts for cited article generation.
Source facts
- Repository
- Tyler-R-Kendrick/tk-technews
- Last source activity
- May 20, 2026 at 22:58
- Detected SKILL.md language
- English
- Stars
- 0
- Forks
- 0
Install options
The review-first prompt is selected by default. You can switch to a direct command or download a local copy.
Review the source files
Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.
Showing SKILL.md
SKILL.md
Source instructions · Read-only preview- name
- technews-webscrape
- description
- Configure and use reputable web scraping for TK TechNews. Use when an agent needs Firecrawl MCP, Firecrawl-backed scraping, dynamic page extraction, web source troubleshooting, or guidance on when to use RSS, local scraping, Firecrawl, or YouTube transcripts for cited article generation.
# TechNews Webscrape
Use this skill from the repository root when a web source needs higher-quality
extraction than the default local `fetch` plus `cheerio` scraper.
## Preferred Scraper
Use Firecrawl as the reputable external scraper. The repo has:
- `.mcp.json` for MCP-aware agents: `npx -y firecrawl-mcp`.
- `npm run scrape:firecrawl -- --url <url>` for one-off extraction checks.
- `data/sources.json` support for `"scraper": "firecrawl"` on web sources.
## Workflow
1. Confirm `FIRECRAWL_API_KEY` is set before expecting MCP or Firecrawl scraping to work.
2. For one URL, run `npm run scrape:firecrawl -- --url "https://example.com/article"`.
3. For configured sources, add `"scraper": "firecrawl"` to the source and run `npm run ingest`.
4. Inspect `data/summaries/latest.json` and `data/raw/firecrawl/` for extraction quality.
5. Run `npm run extract:knowledge`, `npm run draft -- --topic "..."`, and `npm run validate:citations`.
## Selection Rules
- Prefer RSS when the publisher offers a stable feed.
- Use local scraping for simple static pages.
- Use Firecrawl for dynamic pages, pages with heavy chrome, or pages where local extraction misses main content.
- Use YouTube transcript ingestion for videos.
- Do not use scraping to bypass paywalls, access controls, or publisher restrictions.
View on GitHub