| name | blogwatcher |
| description | Monitor blogs and RSS/Atom feeds for updates using the blogwatcher-cli tool. Add blogs, scan for new articles, track read status, and filter by category. |
| version | 2.4.0 |
| author | JulienTant (fork of Hyaxia/blogwatcher), updated with CN network workarounds |
| license | MIT |
| metadata | {"hermes":{"tags":["RSS","Blogs","Feed-Reader","Monitoring"],"homepage":"https://github.com/JulienTant/blogwatcher-cli"}} |
| prerequisites | {"commands":["blogwatcher-cli"],"pip_packages":["feedparser"]} |
Blogwatcher
Track blog and RSS/Atom feed updates with the blogwatcher-cli tool. Supports automatic feed discovery, HTML scraping fallback, OPML import, and read/unread article management.
Installation
curl -sL "https://github.com/JulienTant/blogwatcher-cli/releases/latest/download/blogwatcher-cli_darwin_arm64.tar.gz" -o /tmp/bw.tar.gz
tar xzf /tmp/bw.tar.gz -C /tmp/
sudo cp /tmp/blogwatcher-cli /usr/local/bin/
If downloads fail behind CDN/GWF, retry later or use gh release download.
❌ NEVER Run blogwatcher-cli scan (Unqualified)
Confirmed behavior: blogwatcher-cli scan (without arguments) hangs indefinitely when any Substack/FeedBurner feeds are registered, even though individual scans of those same feeds don't run. The internal worker pool deadlocks on the problem feeds.
Always scan individually by exact registered name:
blogwatcher-cli scan "Google DeepMind Blog" && \
blogwatcher-cli scan "One Useful Thing" && \
blogwatcher-cli scan "Netflix Tech Blog" && \
blogwatcher-cli scan "CrewAI Blog" && \
blogwatcher-cli scan "Interconnects" && \
blogwatcher-cli scan "AWS China Blog" && \
blogwatcher-cli scan "AWS China ML"
Confirmed registered names (2026-07-01, 8 active — Netflix Tech Blog reactivation now permanent): Google DeepMind Blog, One Useful Thing, Netflix Tech Blog, CrewAI Blog, Interconnects, AWS China Blog, AWS China ML, Hugging Face Blog, Stochastic Parrot. 7 good feeds + 2 problem feeds. Netflix reactivation permanent (v5.9.23.35 forward, 6+ consecutive crons) — feed 200, articles 403 via curl, recovery captures 9 articles (10-36KB each) per sweep. Any other spelling will return Error: blog 'X' not found. Run blogwatcher-cli blogs to get the canonical list.
🚨 Critical Pitfall: Proxy Environment Variables in Cron
In the Hermes cron environment, proxy variables cause blogwatcher-cli to fail with authorization errors:
failed to fetch feed: Get "https://deepmind.google/blog/rss.xml":
proxyconnect tcp: dial tcp 127.0.0.1:10808: tcp4/127.0.0.1:10808 is not authorized by the client
Fix: Always unset proxy variables before running blogwatcher-cli:
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY
blogwatcher-cli scan "Feed Name"
This is required for ALL blogwatcher-cli commands in cron jobs. The proxy configuration is inherited from the system but the Go HTTP client doesn't handle local loopback proxies correctly.
🚨 Critical Pitfall: China Network (Substack/FeedBurner/HuggingFace)
blogwatcher-cli (Go HTTP client) hangs on multiple feed types in mainland China:
- Substack (
*.substack.com/feed) — Go net/http hangs 90-120s then times out. Verified: Stochastic Parrot individual blogwatcher-cli scan also times out (2026-05-13), not just batch blogwatcher-cli scan.
- Hugging Face Blog (
huggingface.co/blog/feed.xml) — RSS URL is valid (811 entries, confirmed 2026-06-30), but Go HTTP client times out in CN. Python urllib fetches it in 2-3s.
Confirmed timeout feeds (2026-05-13):
blogwatcher-cli scan "Hugging Face Blog" → 90s+ timeoutblogwatcher-cli scan "Stochastic Parrot" → 60s+ timeout
Fix: Two-track strategy
| Track | Feeds | Method | Works in China? |
|---|
| Good feeds | Non-Substack/non-FuggingFace | blogwatcher-cli scan "Name" individually | ✅ ~2s each |
| Problem feeds | Substack + HuggingFace Blog | Python companion script scripts/fetch-problem-feeds.py | ✅ via urllib + feedparser (2-3s) |
Problem feeds list (always skip in blogwatcher-cli cron):
- Stochastic Parrot (
stochasticparrot.substack.com/feed)
- Hugging Face Blog (
huggingface.co/blog/feed.xml)
These are handled by the rss-problem-feeds-to-inbox cron job (Python, every 6h).
Root cause: Go's net/http TLS/TCP stack behavior against certain Chinese CDN/proxy infrastructure. Python's urllib.request with full Chrome UA fetches these same feeds in 2-3s.
Substack URL verification pitfall (2026-05-14): curl -sL -I --max-time 10 on Substack article URLs returns zero output (connection hangs silently — no HTTP status line, no timeout error). This makes curl -I unreliable for Substack URL health checks. In practice, rely on file existence + size in rss-inbox as the signal of successful fetch, not curl status line.
🚨 Self-Update Pattern: Avoid blogwatcher-cli scan (unqualified)
Always scan feeds individually by name. This session confirmed that blogwatcher-cli scan (without arguments) hangs indefinitely due to problem feeds. Individual scans (blogwatcher-cli scan "Latent Space") complete in ~2s per feed.
For a full set of feeds, chain them:
blogwatcher-cli scan "Feed A" && blogwatcher-cli scan "Feed B" && blogwatcher-cli scan "Feed C"
🚨 Shell timeout Command Missing on macOS (verified 2026-06-18)
Symptom: Cron scripts that work on Linux fail on macOS with timeout: command not found because macOS does NOT ship GNU coreutils (no timeout, gtimeout, realpath, sha256sum etc. by default).
Fixes (in order of preference):
- Best — Python wrapper (already documented in rss-to-wiki-pipeline):
subprocess.run(['blogwatcher-cli', 'scan', feed], timeout=20). This is what production cron scripts use.
- Pure-bash fallback (works without GNU coreutils):
(cmdpid=$BASHPID; (sleep N; kill $cmdpid 2>/dev/null) &) ; <command> — runs a parallel sleep+killer. Ugly but works.
- perl alarm wrapper (verified 2026-06-18 02:58):
/usr/bin/perl -e 'alarm N; exec("argv", "arg1", ...)' — every macOS ships perl. Use this when you need a hard outer timeout on a non-Python script and don't want to write a wrapper.
- Install gtimeout via Homebrew:
brew install coreutils then gtimeout works. Not recommended for cron — adds a Homebrew dependency to the production pipeline.
Why this matters for blogwatcher-cli specifically: blogwatcher-cli's Go HTTP client can hang indefinitely on Substack/Hugging Face feeds in CN. If you forget to use a per-feed timeout wrapper, one hung scan blocks the entire cron run.
🚨 CDN 403 for Minimal User-Agent
Several feeds (Latent Space, Interconnects, Ben's Bites, Algo Bridge) return HTTP 403 when fetched with bare User-Agent: Mozilla/5.0. They work with Chrome's full User-Agent string. This affects both blogwatcher-cli (Go) and Python if the header is too minimal.
This was the root cause of the "no new articles" / "scan succeeds but 0 new" false negative that persisted across multiple scans. blogwatcher-cli uses a minimal UA by default.
Fix (in companion script): Always use the full Chrome user agent.
The companion Python script already uses the full Chrome UA list from this lesson.
🚨 308 Permanent Redirects Are NOT Followed by Go HTTP Client
Symptom (verified 2026-06-15): Feed at https://psyche.co/feed returns HTTP/2 308 Permanent Redirect → /feed.rss. blogwatcher-cli scan reports failed to fetch feed: status 308 and exits non-zero. Go's net/http does not auto-follow 308.
This is not limited to 308 — any non-standard redirect that the Go client doesn't recognize will fail the same way. Common offenders:
/feed → /feed.rss (Psyche, Aeon sister sites)/rss → /feed.xml- Path migrations during site rebuilds
Fix: Always verify canonical URL with curl BEFORE adding the feed:
curl -sLI --max-time 15 -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36" \
https://example.com/feed | grep -iE "^(HTTP|location)"
If the response includes a 3xx with a location: header, use the redirected URL in add --feed-url, not the original.
🚨 Podcast RSS Feeds Have No Transcripts (Verified 2026-06-15)
Symptom: Adding a podcast RSS feed (Farnam Street, The Knowledge Project, etc.) succeeds. scan reports 20 "new articles". But every entry is only <title> + <url> — there is no <description> with transcript text, no full content. The RSS spec for podcast feeds intentionally excludes transcripts.
Why this bites content extraction:
- A normal blog RSS gives you
title + url + summary (often 200-500 words of preview)
- A podcast RSS gives you
title + url and that's it
- To get content you must crawl the episode page, find transcript block (often paywalled or absent), then scrape — 5-10x the token cost of a normal blog
- Many "podcast episodes" of Aeon/Psyche are video essays with
category=video — they have a 1-paragraph description, no transcript
Decision rule: BEFORE adding a podcast feed, ask:
- Does the publisher provide public transcripts? (Farnam Street: no for free tier. Tim Ferriss: yes. Lex Fridman: yes.)
- If no transcript, do you accept that "ingest" becomes a 2-step pipeline (RSS scan → playwright scrape → MD write)?
- Are episodes likely to overlap with another vault's existing entities? (Farnam Street regularly features Greg Brockman, AI infra CEOs → will trip cross-vault dedup.)
If answer is "no transcript + don't want scrape step + overlap risk" → don't add the podcast feed as RSS. Treat it as a one-off manual ingest, or skip.
Verified case (2026-06-15): Added Farnam Street to wiki-life RSS DB. Scan returned 20 episodes, but ~10 were tech-domain (Greg Brockman, AI infra) and would all be rejected by cross-vault dedup against ~/wiki/raw/articles/. Lesson: filter podcasts by episode title keywords before adding, not after.
Multi-DB Isolation for Dual Vaults
If you run more than one content vault (e.g. ~/wiki for AI/ML research and ~/wiki-life for personal growth), never let them share the same blogwatcher DB. Cross-pollution: scanning a tech feed from "life" cron triggers phantom re-reads, double-counts unread articles, and breaks the "I read it in tech, it shows up again in life inbox" failure mode.
Pattern: --db per vault. Full recipe and pre-flight checklist: see references/dual-vault-isolation.md.
WIKI_DB=/Users/jinguo/.blogwatcher-cli/blogwatcher-cli.db
WIKI_BLOGS="Google DeepMind Blog" "One Useful Thing" "CrewAI Blog" \
"Interconnects" "AWS China Blog" "AWS China ML"
LIFE_DB=/Users/jinguo/.blogwatcher-cli/blogwatcher-cli-life.db
LIFE_BLOGS="少数派" "Cal Newport" "Farnam Street" "Psyche (Aeon)"
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY
for feed in "${WIKI_BLOGS[@]}"; do
blogwatcher-cli --db "$WIKI_DB" scan "$feed"
done
Pitfall: DB created on first add. If the DB file doesn't exist, the first add initializes it. Don't pre-touch the file expecting "an empty DB" — let add do it.
Pitfall: cron job isolation. Each cron job's prompt MUST pin --db $DB_PATH explicitly. If a cron forgets the flag, it falls back to the default ~/.blogwatcher-cli/blogwatcher-cli.db and silently crosses the vault boundary.
Pitfall: profile name uniqueness. Even with separate DBs, feed names are stored per-DB. A blog named "Farnam Street" can exist in both DBs — they don't collide. So naming uniqueness is NOT a problem.
🚨 Retired Feed Cleanup Pitfall
When a feed is retired (e.g., blogwatcher-cli remove "Name"), ALL scan chains must be updated immediately. The blogwatcher skill's scan chain and the rss-to-wiki-pipeline cron instructions both reference the active feed list. If only the DB removal is done but the scan chain isn't updated, the next cron run will attempt blogwatcher-cli scan "Retired Feed" which either:
- Returns
Error: blog 'X' not found (non-fatal but pollutes logs)
- Times out at 30s if the feed was removed from DB but the name still resolves
Verified 2026-06-21: Netflix Tech Blog was retired 2026-06-16 but the scan chain still included it, causing 30s timeouts on every cron run. Fix: update BOTH skills AND companion Python scripts when retiring a feed.
3-location update checklist when retiring a feed:
blogwatcher-cli remove "Feed Name" --yes — remove from DB- Update skill SKILL.md tables (blogwatcher + rss-to-wiki-pipeline) — remove from feed lists
- Update companion Python scripts'
GOOD_FEEDS dicts (rss-inbox-watchdog.py, fetch-problem-feeds.py, rss-inbox-recovery.py, rss-inbox-curl-recovery.py) — remove from feed URL maps
v5.9.23.33 verified: rss-inbox-watchdog.py still had "Netflix Tech Blog": "https://netflixtechblog.com/feed" in GOOD_FEEDS 5 days after retirement. The watchdog's hardcoded "/9" feed count was also stale (should auto-resolve via len(GOOD_FEEDS)).
🚨 Watchdog Processing Gap for Problem Feeds
Architecture issue (discovered 2026-06-27, partially addressed in v3, but DB marking still broken): The companion scripts had a processing gap where fetch-problem-feeds.py (v1, now removed from blogwatcher skill) inserted articles into blogwatcher DB with is_read=0, but the watchdog never processed them.
v3 attempt: The current script at ~/wiki/scripts/fetch-problem-feeds.py writes MD to raw/rss-inbox/ AND contains code to mark articles as is_read=1 in the DB via UPDATE articles SET is_read=1 WHERE url=? AND is_read=0.
⚠️ Observed behavior (consistent across 5+ sessions, 2026-07-02/03): The script consistently reports blogwatcher DB: 0 articles marked as read. Root cause: URL format mismatch between the RSS feed entry.link and blogwatcher's articles.url column. Common differences include trailing slashes, URL encoding, and protocol variants (https vs http). The UPDATE statement uses exact URL matching, so it never finds a matching row.
Impact: The unread count for problem feeds (Stochastic Parrot, Hugging Face Blog) in blogwatcher DB grows indefinitely. The inbox dedup logic in fetch-problem-feeds.py prevents duplicate writes, so pipeline operation is not affected — the only consequence is stale unread counts in blogwatcher-cli articles.
Manual cleanup — still needed periodically:
sqlite3 ~/.blogwatcher-cli/blogwatcher-cli.db "UPDATE articles SET is_read = 1 WHERE is_read = 0;"
Run after scan cycles when unread count seems inflated. This is safe — it marks ALL unread as read, and the pipeline only cares about new articles from RSS, not the DB's read state.
Current Cron Setup (as of 2026-06-30 — Netflix Reactivated)
The old dual-cron setup (C1: scan good feeds + C2: full scan) was replaced. The current 4-cron architecture delegates content conversion to rss-to-wiki-pipeline's Python scripts. blogwatcher-cli's role is now limited to scanning and tracking new article titles+URLs — content extraction is handled separately.
See rss-to-wiki-pipeline skill for the full cron configuration. The Hermes crons are:
| Cron Name | Frequency | What It Does |
|---|
rss-good-feeds-to-inbox | Every 4h | Scan 7 active feeds (blogwatcher-cli) + rss-inbox-watchdog → raw/rss-inbox/ |
rss-problem-feeds-to-inbox | Every 6h | Python fetch Substack/HuggingFace → raw/rss-inbox/ |
rss-inbox-weekly-promote-cleanup | Mon 10:00 | Cleanup + promote screening |
rss-source-health-audit | Mon 11:00 | Feed health check + auto-block failing feeds |
Registered feeds (10 total, as of 2026-07-02 — +Import AI detected in DB):
- Good feeds (7): Google DeepMind Blog, One Useful Thing, Netflix Tech Blog (permanent — 9 articles 10-36KB per sweep), CrewAI Blog, Interconnects, AWS China Blog, AWS China ML
- Problem feeds (2): Hugging Face Blog, Stochastic Parrot (require Python fallback)
- Pending (1): Import AI (
importai.substack.com/feed) — newly registered in blogwatcher DB but NOT in any pipeline script. Needs decision: add to fetch-problem-feeds.py or remove from blogwatcher.
- Netflix Tech Blog REACTIVATED 2026-06-30, PERMANENT (v5.9.23.35 forward, 6+ consecutive crons): Feed URL (
netflixtechblog.com/feed) returned to life after ~2 weeks of 404s. Individual article URLs return HTTP 403 (medium.com CDN blocking direct curl), but the recovery script captures full content from feed entries (10-36KB each). Signal pattern: feed 200 + articles 403 via curl + recovery works = content is valid. Include in scan chains and recovery scripts.
Common Commands
blogwatcher-cli add "Name" https://example.com --feed-url https://example.com/feed.xml
blogwatcher-cli remove "Name" --yes
blogwatcher-cli blogs
blogwatcher-cli scan "Feed Name"
blogwatcher-cli articles
blogwatcher-cli articles --blog "Name"
blogwatcher-cli read 1
blogwatcher-cli read-all --yes
RSS Feed Discovery Reference
AWS Blogs
Pattern: https://aws.amazon.com/blogs/<category>/feed/
All 15 categories work in China. Recommended subset for tech focus: Architecture, Big Data, Compute, Database, DevOps, Developer, Infrastructure, ML, Networking, Open Source, Security, Storage.
| Blog Category | Feed URL | Notes |
|---|
| Architecture | https://aws.amazon.com/blogs/architecture/feed/ | ✅ |
| Big Data | https://aws.amazon.com/blogs/big-data/feed/ | ✅ |
| Compute | https://aws.amazon.com/blogs/compute/feed/ | ✅ |
| Database | https://aws.amazon.com/blogs/database/feed/ | ✅ |
| DevOps | https://aws.amazon.com/blogs/devops/feed/ | ✅ |
| Developer | https://aws.amazon.com/blogs/developer/feed/ | ✅ |
| Infrastructure | https://aws.amazon.com/blogs/infrastructure-and-automation/feed/ | ✅ |
| ML (Machine Learning) | https://aws.amazon.com/blogs/machine-learning/feed/ | ✅ |
| Networking | https://aws.amazon.com/blogs/networking-and-content-delivery/feed/ | ✅ |
| Open Source | https://aws.amazon.com/blogs/opensource/feed/ | ✅ |
| Security | https://aws.amazon.com/blogs/security/feed/ | ✅ |
| Storage | https://aws.amazon.com/blogs/storage/feed/ | ✅ |
| Containers | https://aws.amazon.com/blogs/containers/feed/ | ⚠️ 10 entries (fewer updates) |
| Industries | https://aws.amazon.com/blogs/industries/feed/ | ⚠️ Industry vertical, low tech depth |
| Startups | https://aws.amazon.com/blogs/startups/feed/ | ⚠️ 400 entries but startup news, low depth |
AI Labs
| Source | Feed URL | Notes |
|---|
| OpenAI Blog | https://openai.com/news/rss.xml | Redirects from blog/rss.xml — verified 940 entries |
| Google DeepMind | https://deepmind.google/blog/rss.xml | 100 entries, works directly |
| Anthropic | — | No public RSS feed (checked all patterns) |
| Meta AI | — | No public RSS feed (checked all patterns) |
Independent AI Sources (Good)
| Source | Feed URL | Status |
|---|
| Latent Space | https://www.latent.space/feed.xml | ❌ Removed (newsletter digest, 2026-05) |
| One Useful Thing | https://www.oneusefulthing.org/feed | ✅ Active |
| Interconnects | https://www.interconnects.ai/feed | ✅ Active |
| Last Week in AI | https://lastweekin.ai/feed | ❌ Removed (newsletter digest) |
| Cloudflare AI | https://blog.cloudflare.com/rss/ | ❌ Removed (product announcements) |
| Netflix Tech Blog | https://netflixtechblog.com/feed | ✅ REACTIVATED 2026-06-30, PERMANENT (feed 200+; articles 403 via curl but recovery captures 9+ articles 10-36KB per sweep) |
| BAIR (Berkeley) | https://bair.berkeley.edu/blog/feed.xml | ❌ Removed (low update frequency) |
| SAIL (Stanford) | https://ai.stanford.edu/blog/feed.xml | ❌ Removed (summary-only) |
| CrewAI Blog | https://blog.crewai.com/rss/ | ✅ Active |
| Hugging Face Blog | https://huggingface.co/blog/feed.xml | ⚠️ Problem feed (Go HTTP timeout in CN); use fetch-problem-feeds.py |
| Stochastic Parrot | https://stochasticparrot.substack.com/feed | ⚠️ Problem feed (Substack + Go HTTP timeout); use fetch-problem-feeds.py |
Substack Feeds (Problem — use Python script)
| Source | Feed URL |
|---|
| Stochastic Parrot | https://stochasticparrot.substack.com/feed |
301 Redirect Verification
Always verify feed URLs before adding. Many sources have moved domains:
curl -sL -o /dev/null -w "%{url_effective}" <feed_url>
echo
Known redirects (verified in 2026-05):
bensbites.com/feed → www.bensbites.com/feedwww.strictlyvc.com/feed → strictlyvc.com/feed/thealgorithmicbridge.substack.com/feed → www.thealgorithmicbridge.com/feedopenai.com/blog/rss.xml → openai.com/news/rss.xml
Companion Script
The fetch-problem-feeds.py script is maintained by the rss-to-wiki-pipeline skill. Canonical location: ~/wiki/scripts/fetch-problem-feeds.py (verified 2026-07-02) — v3 writes MD directly to raw/rss-inbox/ and attempts to mark blogwatcher DB articles as read, but see "Watchdog Processing Gap" section above for the known URL mismatch issue that prevents DB marking from actually working.
Invoke standalone (from anywhere):
/usr/bin/python3 ~/wiki/scripts/fetch-problem-feeds.py
/usr/bin/python3 ~/wiki/scripts/fetch-problem-feeds.py --dry-run
Database: ~/.blogwatcher-cli/blogwatcher-cli.db
Schema:
CREATE TABLE blogs (id INTEGER PRIMARY KEY, name TEXT NOT NULL, url TEXT NOT NULL UNIQUE, feed_url TEXT, scrape_selector TEXT, last_scanned TIMESTAMP);
CREATE TABLE articles (id INTEGER PRIMARY KEY, blog_id INTEGER NOT NULL, title TEXT NOT NULL, url TEXT NOT NULL UNIQUE, published_date TIMESTAMP, discovered_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP, is_read BOOLEAN DEFAULT FALSE, categories TEXT);
User-Agent Requirements
Many RSS feeds behind Cloudflare return 403 for minimal User-Agent Mozilla/5.0. Use a full Chrome UA:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36
This is already baked into the companion script.
Notes
- Auto-discovers RSS/Atom feeds from blog homepages (unreliable behind CDN — always use
--feed-url)
- Falls back to HTML scraping when
--scrape-selector is configured
- Import from OPML:
blogwatcher-cli import subscriptions.opml
