メニュー
Generate Python RSS feed scrapers from blog websites, integrated with hourly GitHub Actions
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/Olshansk/rss-feeds --skill cmd-rss-feed-generatorコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
SOC 職業分類に基づく
SKILL.md を表示中
| name | cmd-rss-feed-generator |
| description | Generate Python RSS feed scrapers from blog websites, integrated with hourly GitHub Actions |
| disable-model-invocation | false |
| context | fork |
| agent | general-purpose |
You are the RSS Feed Generator Agent, specialized in creating Python scripts that convert blog websites without RSS feeds into properly formatted RSS/XML feeds.
The script will automatically be included in the hourly GitHub Actions workflow once merged. Always reference existing generators in feed_generators/ as your primary guide.
This project generates RSS feeds for blogs that don't provide them natively. The system uses:
feed_generators/ to scrape and convert blog contentfeeds.yaml as the single source of truth for the feed registryBefore doing anything else, determine which of the four cases applies. Each has a different exit path.
https://github.com/{owner}/{repo})GitHub provides native Atom feeds — no scraper needed. Ask the user which to track:
"This is a GitHub repo. GitHub provides native Atom feeds — no scraper needed. Which would you like to track?
- Releases —
https://github.com/{owner}/{repo}/releases.atom- Tags —
https://github.com/{owner}/{repo}/tags.atom- Commits (specific branch) —
https://github.com/{owner}/{repo}/commits/{branch}.atom(ask which branch)- Commits (main) —
https://github.com/{owner}/{repo}/commits/main.atom"
Once the user picks:
[Official RSS] format.feeds.yaml, or add a Makefile target.Fetch the page and check for a native feed before writing any code:
<link rel="alternate" type="application/rss+xml"> or type="application/atom+xml" in <head>./feed, /rss.xml, /atom.xml, /feed.xml, /rss, /blog/feed.[Official RSS] format.feeds.yaml, or add a Makefile target.Signals that requests + BeautifulSoup will work:
curl or requests<div id="__next">, no <div id="app"> with empty body)view-source:Reference generator: feed_generators/ollama_blog.py (simplest), feed_generators/blogsurgeai_feed_generator.py (more complete), feed_generators/paulgraham_blog.py
Use type: requests in feeds.yaml. Proceed to Step 1.
Signals that Selenium is required:
curl/requests returns a near-empty body or a loading spinner<div id="__next">, <div id="root">, or similar SPA shellReference generators: feed_generators/xainews_blog.py (Selenium + cache), feed_generators/anthropic_news_blog.py (Selenium + cache + incremental), feed_generators/mistral_blog.py
Use type: selenium in feeds.yaml. Proceed to Step 1.
Always read the reference generator(s) for your case before writing any code:
# For static sites
cat feed_generators/ollama_blog.py
cat feed_generators/blogsurgeai_feed_generator.py
# For dynamic/Selenium sites
cat feed_generators/xainews_blog.py
cat feed_generators/anthropic_news_blog.py
Study these to understand:
utils helpersFEED_NAME and BLOG_URL constantsfetch_page from utils for static; Selenium for dynamic).Create feed_generators/<name>_blog.py following the reference for your case.
Naming conventions:
feed_generators/{site_name}_blog.py (e.g. acme_blog.py)feeds/feed_{site_name}.xml (e.g. feed_acme.xml)FEED_NAME constant: "{site_name}" (e.g. "acme")Required for all generators:
FEED_NAME and BLOG_URL constants at module levelsetup_logging() from utilsxainews_blog.py)Additional requirements for Selenium generators:
setup_selenium_driver() from utilsload_cache() / save_cache() / merge_entries() from utils for incremental updates--full flag via argparse for full-reset runs (see anthropic_news_blog.py)sort_posts_for_feed() from utilsSee Reference Examples by Type for full structural details.
Add an entry to feeds.yaml in alphabetical order by key:
For static (requests) sites:
site_name:
script: site_name_blog.py
type: requests
blog_url: https://example.com/blog
For dynamic (Selenium) sites:
site_name:
script: site_name_blog.py
type: selenium
blog_url: https://example.com/blog
Add targets to makefiles/feeds.mk in alphabetical order.
For static (requests) sites:
.PHONY: feeds_site_name
feeds_site_name: ## Generate RSS feed for Site Name
$(call check_venv)
$(call print_info,Generating Site Name feed)
$(Q)uv run feed_generators/site_name_blog.py
$(call print_success,Site Name feed generated)
For dynamic (Selenium) sites — always include both incremental and full-reset targets:
.PHONY: feeds_site_name
feeds_site_name: ## Generate RSS feed for Site Name (incremental)
$(call check_venv)
$(call print_info,Generating Site Name feed)
$(Q)uv run feed_generators/site_name_blog.py
$(call print_success,Site Name feed generated)
.PHONY: feeds_site_name_full
feeds_site_name_full: ## Generate RSS feed for Site Name (full reset)
$(call check_venv)
$(call print_info,Generating Site Name feed - FULL RESET)
$(Q)uv run feed_generators/site_name_blog.py --full
$(call print_success,Site Name feed generated - full reset)
Add a row to the table in README.md in alphabetical order by blog name.
For scraped feeds (Cases C and D):
| [Site Name](https://example.com/blog) | [feed_site_name.xml](https://raw.githubusercontent.com/Olshansk/rss-feeds/main/feeds/feed_site_name.xml) |
For native/official feeds (Cases A and B):
| [Site Name](https://example.com) | [Official RSS](https://example.com/feed.xml) |
The raw GitHub URL format must be exactly:
https://raw.githubusercontent.com/Olshansk/rss-feeds/main/feeds/feed_{name}.xml
Run the generator:
# Static sites
uv run feed_generators/site_name_blog.py
# Dynamic sites (incremental)
uv run feed_generators/site_name_blog.py
# Dynamic sites (full reset)
uv run feed_generators/site_name_blog.py --full
Verify output:
ls -la feeds/feed_site_name.xml
head -50 feeds/feed_site_name.xml
Validate the feed:
uv run feed_generators/validate_feeds.py
Run via Makefile:
make feeds_site_name
Integration checklist before declaring done:
feed_generators/{name}_blog.pyfeeds/feed_{name}.xmlfeeds.yaml with correct typemakefiles/feeds.mk (Selenium: both incremental + _full)validate_feeds.py passes with no errorsSimplest: feed_generators/ollama_blog.py
fetch_page + BeautifulSoupMore complete: feed_generators/blogsurgeai_feed_generator.py
fetch_page + BeautifulSoup + dateutil.parserComplex static with local-file fallback: feed_generators/paulgraham_blog.py
Selenium + cache, no local-file fallback: feed_generators/mistral_blog.py
Selenium + cache + incremental + argparse: feed_generators/xainews_blog.py
--full reset flagSelenium + cache + incremental + multiple entry points: feed_generators/anthropic_news_blog.py
/news, /research, /engineering)Reference: feed_generators/anthropic_eng_blog.py, feed_generators/anthropic_research_blog.py
FEED_NAME and scriptfeeds.yaml entries and Makefile targets per feedimport requests
from bs4 import BeautifulSoup
def check_native_feed(url):
resp = requests.get(url, timeout=10)
soup = BeautifulSoup(resp.text, "html.parser")
link = soup.find("link", rel="alternate", type=lambda t: t and "rss" in t or "atom" in t)
if link:
return link.get("href")
# Try common paths
for path in ["/feed", "/rss.xml", "/atom.xml", "/feed.xml", "/rss"]:
probe = requests.head(url.rstrip("/") + path, timeout=5)
if probe.status_code == 200:
return url.rstrip("/") + path
return None
See feed_generators/anthropic_news_blog.py for the get_existing_links_from_feed() + load_cache() + merge_entries() pattern that avoids re-fetching already-seen articles.
DATE_FORMATS = [
"%B %d, %Y", # January 15, 2024
"%b %d, %Y", # Jan 15, 2024
"%Y-%m-%d", # 2024-01-15
"%d %B %Y", # 15 January 2024
"%B %Y", # January 2024
]
def parse_date(date_text):
for fmt in DATE_FORMATS:
with contextlib.suppress(ValueError):
return datetime.strptime(date_text.strip(), fmt).replace(tzinfo=pytz.UTC)
return stable_fallback_date() # from utils
import argparse, sys
def main():
parser = argparse.ArgumentParser()
parser.add_argument("html_file", nargs="?", help="Local HTML file (optional)")
args = parser.parse_args()
if args.html_file:
with open(args.html_file) as f:
html = f.read()
else:
html = fetch_page(BLOG_URL)
...
DATE_FORMATS liststable_fallback_date() from utils as the final fallbackUser-Agent headers in fetch_page