菜单
精选高价值 RSS feeds 扫描，输出到 raw/rss-inbox/ 暂存区。只保留有独立知识深度的 feed（非 digest 类），不再自动入库。包含 rss-inbox-curl-recovery.py 绕过 blogwatcher read-state 漏抓的兜底。
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/QianJinGuo/wiki --skill rss-to-wiki-pipeline命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
Meta-orchestrator that wires web-content-reviewer, llm-wiki, and wiki-evolver into a single four-phase knowledge pipeline (Triage → Gate → Store → Evolve). Use as the single entry point for all knowledge base operations. Includes a 6-URL-validated user-pasted WeChat URL fast path, three-axis dedup decision matrix (NEW/MERGE/DEDUP), orphan-raw detection protocol, and sibling-subagent race V7 evidence, and a two-variant V6 mid-write fix for matching vs different-slug duplicates.
从 Gmail 中提取 TLDR AI 等 newsletter 的链接，写入 raw/email-inbox/candidates.md，供后续 inbox-screener 评分。只做链接不做评分，与 inbox-screener 配合使用。
Generate periodic status reports for the Obsidian wiki — gather directory stats, discover recently committed entities, extract publish metadata from raw articles, and format a deliverable report. Covers daily/weekly cron-driven automation and ad-hoc health checks.
正在显示 SKILL.md
| name | rss-to-wiki-pipeline |
| description | 精选高价值 RSS feeds 扫描，输出到 raw/rss-inbox/ 暂存区。只保留有独立知识深度的 feed（非 digest 类），不再自动入库。包含 rss-inbox-curl-recovery.py 绕过 blogwatcher read-state 漏抓的兜底。 |
| version | 5.9.24.5 |
| author | Hermes Agent |
| category | wiki |
| related_skills | ["wiki-pipeline","web-content-reviewer"] |
Archived 2026-08-12 maintenance (size mgmt): inline blocks for 2026-08-12 00:56 / 03:21 / 05:36 / 07:41 / 09:52 / 14:33 / 16:58 (runs 1-7) moved to references/skill-changelog-archive-sessions-2026-08-12-0056-to-2026-08-12-1658.md (SKILL.md was at 101,600/100,000 chars — over cap, archive required before next pointer update)
📦 Historical changelogs archived: v5.8.0 → v5.9.22.7 + per-cron session logs are in
references/cron-run-changelog-archive.mdandreferences/cron-run-YYYY-MM-DD-HHMM.md. Older per-version SKILL.md changelogs (v5.9.23.0 → v5.9.23.14) inreferences/skill-changelog-archive-v5.9.23.0-to-v5.9.23.14.md; v5.9.23.15 → v5.9.23.20 inreferences/skill-changelog-archive-v5.9.23.15-to-v5.9.23.20.md; v5.9.23.21 → v5.9.23.29 archived:references/skill-changelog-archive-v5.9.23.21-to-v5.9.23.29.md. v5.9.23.33 → v5.9.23.40 archived:references/skill-changelog-archive-v5.9.23.33-to-v5.9.23.40.md. Older changelog blocks have been removed from this SKILL.md — see the individual session logs for per-version detail. This SKILL.md carries only the most recent session log pointer + Stable Operational Patterns table (the high-signal knowledge a future session needs to read FIRST) + operational steps.
Latest session log: references/cron-run-2026-08-28-0512.md (2026-08-28 05:12, 3rd run of day — FRESH-CYCLE REBUILD, byte-identical deterministic steady state to 03:10 baseline — 0→173 (big=75, mid=66, small=32): start inbox=0 (drained 04:12 wiki-inbox-scan-v2 + 05:06 wechat-inbox-pipeline, both post-03:10 with 0 rss ingest; 03:5x wechat line confirms "rss 173 (03:10 rebuild)... 0 genuinely-new", 05:0x confirms "rss 0 (无新 rebuild)"). 7 std scans: all 7 New=0 except AWS-ML New=1 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): all 11 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=210 written, 105 skipped — byte-identical to 03:10 (std 47: AWS-CB 9, AWS-ML 14, CrewAI 9, Interconnects 8, Netflix 6, OUT 1 + WeChat 163: 量子位 18, 夕小瑶 18, 字节 15, 小米 17, 新智元 18, 机器之心 13, 美团 17, 腾讯 10, 阿里云开发者 15, 阿里技术 10, PaperWeekly 12); watchdog=1 written (AWS-ML, ACTIVE via scan New=1, stale-re-report dup → 0 net); curl-recovery=49 (GDM 49). Filters: dedup_rm=32, case-study_rm=7, sub-1KB-stub_rm=48, trafilatura-upgrade=0 (all byte-identical). Final inbox 0→173 (big=75, mid=66, small=32), 173 unique URLs, 0 intra-inbox dups. Count 0 vs 03:10 baseline 173 — exactly byte-identical on EVERY component (recovery 210, curl 49, HF 5, dedup 32, case-study 7, sub-1KB 48, traf 0, final 173, composition 75/66/32, URL-200 165/173); all scans New=0 (except AWS-ML stale re-report) → no organic content; watchdog short-circuited confirms; no ingestion since 03:10 (both drains 0 rss ingest); zero feed-death signals. Reconciliation: 210 + 49 + 5 (HF) = 264 gross − 32 − 7 − 48 = 177, minus 4 already-ingested-skipped → 173 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD, byte-identical deterministic steady state, NOT stale/feed-health — pure determinism, no delta on any component, no escalation. URL-200=165/173 (aws 23/23, mp.weixin 134/134, interconnects 7/7, oneusefulthing 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy).
Prior session log: references/cron-run-2026-08-28-0306.md (2026-08-28 03:06, 2nd run of day — FRESH-CYCLE REBUILD with ingestion-driven reverse shift (−3) — 0→173 (big=75, mid=66, small=32): start inbox=0 (drained ~02:07 wiki-inbox-scan-v2 + 02:52 wechat-inbox-pipeline, both post-01:04 rebuild with 0 rss ingest since; 02:49 wechat line confirms "rss 0 (无新 rebuild)"). 7 std scans: all 7 New=0 except AWS-ML New=1 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): all 11 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=210 written, 105 skipped — byte-identical to 01:04 (std 47: AWS-CB 9, AWS-ML 14, CrewAI 9, Interconnects 8, Netflix 6, OUT 1 + WeChat 163: 量子位 18, 夕小瑶 18, 字节 15, 小米 17, 新智元 18, 机器之心 13, 美团 17, 腾讯 10, 阿里云开发者 15, 阿里技术 10, PaperWeekly 12); watchdog=1 written (AWS-ML, ACTIVE via scan New=1, dup → 0 net — stale-DB re-report); curl-recovery=49 (GDM 49). Filters: dedup_rm=32 (baseline 29 +3 = the 3 ingested), case-study_rm=7, sub-1KB-stub_rm=48, trafilatura-upgrade=0 (case-study/sub-1KB/traf byte-identical). Final inbox 0→173 (big=75, mid=66, small=32), 173 unique URLs, 0 intra-inbox dups. Count −3 vs 01:04 baseline 176 — decomposed: the −3 = exactly the 3 rss-inbox articles ingested by the 01:4x wechat-inbox-pipeline (OmniColor ECCV + GRACE ICML + TrAct 李飞飞, all verified ingested: 2026-08-28 in raw/articles with mp.weixin source_urls), now removed by dedup (29→32 +3) despite byte-identical recovery 210. Composition 76/68/32 → 75/66/32 (−1 big −2 mid maps to the 3 ingested). Reconciliation: 210 + 49 + 5 (HF) = 264 gross − 32 − 7 − 48 = 177, minus 4 already-ingested-skipped → 173 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD with ingestion-driven reverse shift (−3), NOT stale/feed-health — the −3 is decomposed exactly to the 3 ingests; zero feed-death signals (all std + curl HTTP=200). URL-200=165/173 (aws 23/23, mp.weixin 134/134, interconnects 7/7, OUT 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy, no escalation).
Prior session log: references/cron-run-2026-08-28-0104.md (2026-08-28 01:04, 1st run of day — FRESH-CYCLE REBUILD with organic multi-feed cascade + ingestion-driven 新智元 reverse shift — 0→176 (big=76, mid=68, small=32): start inbox=0 (drained 00:00 wiki-inbox-scan-v2 + 00:36 wechat-inbox-pipeline, both post-18:44 with 0 rss ingest; 00:25 ingests alibaba-architect-agent / ruofei-ai-native-sdlc / distributed-rl were WeChat-大模型智能-架构师-阿里技术 path, not rss-inbox). 7 std scans: GDM New=2, AWS-ML New=2, other 5 New=0 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): 机器之心 New=3, 美团 New=3, 阿里技术 New=2, PaperWeekly New=2, 腾讯 New=1, 字节 New=1, 夕小瑶 New=1, other 4 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=210 written, 105 skipped — std 47 (AWS-CB 9, AWS-ML 14, CrewAI 9, Interconnects 8, Netflix 6, OUT 1) + WeChat 163 (量子位 18, 夕小瑶 18, 字节 15, 小米 17, 新智元 18 [19→18 −1 Ornith-1.5 ingestion], 机器之心 13, 美团 17, 腾讯 10, 阿里云开发者 15, 阿里技术 10 [9→10 +1 organic], PaperWeekly 12 [11→12 +1 organic]); watchdog=2 written (AWS-ML, ACTIVE via scan New=2 — organic corroboration); curl-recovery=49 (GDM 49). Filters: dedup_rm=29 (baseline 32 — 3 fewer dups as 3 organic new re-enter), case-study_rm=7, sub-1KB-stub_rm=48 (baseline 49), trafilatura-upgrade=0. Final inbox 0→176 (big=76, mid=68, small=32), 176 unique URLs, 0 intra-inbox dups. Count +4 vs 18:44 baseline 172 — decomposed: (1) AWS-ML organic +2 (scan New=2, watchdog ACTIVE wrote 2, all ~15-31KB big); (2) 阿里技术 +1 + PaperWeekly +1 organic (recovery 9→10 / 11→12, scan New=2 each — confirmed organic, not stale re-report, watchdog ACTIVE); (3) 新智元 −1 = ingestion-driven reverse shift (Ornith-1.5 self-play self-generated coding RL ingested 19:00, verified feed_name: WeChat-新智元 in raw/articles, now skipped 19→18). Composition 74/65/33 → 76/68/32 (+2 big +3 mid −1 small). Reconciliation: 210 + 49 + 5 (HF) = 264 gross − 29 − 7 − 48 = 180, minus 4 already-ingested-skipped → 176 clean (watchdog net +2 organic). Classification: FRESH-CYCLE REBUILD with organic multi-feed cascade (+2 AWS-ML +1 阿里技术 +1 PaperWeekly = +4) + ingestion-driven 新智元 reverse shift (−1), NOT stale/feed-health — the +4 is decomposed exactly to the organic set; zero feed-death signals (all std + curl HTTP=200). URL-200=168/176 (aws 23/23, mp.weixin 137/137, interconnects 7/7, oneusefulthing 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy, no escalation).
Prior session log: references/cron-run-2026-08-27-1844.md (2026-08-27 18:44, 8th run of day — FRESH-CYCLE REBUILD with organic WeChat-新智元 cascade (+1 big) — 0→172 (big=74, mid=65, small=33): start inbox=0 (drained 17:39 wiki-inbox-scan-v2 + 17:56 wechat-inbox-pipeline, both post-16:20 with 0 rss ingest). 7 std scans: all 7 New=0 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): 新智元 New=3, other 10 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=209 written, 106 skipped — std 47 byte-identical to 16:20 (AWS-CB 9, AWS-ML 14, CrewAI 9, Interconnects 8, Netflix 6, OUT 1) + WeChat 162 (量子位 18, 夕小瑶 18, 字节 15, 小米 17, 新智元 19 [18→19 +1 organic], 机器之心 13, 美团 17, 腾讯 10, 阿里云开发者 15, 阿里技术 9, PaperWeekly 11); watchdog=0 written (GDM short-circuit, all std scans New=0); curl-recovery=50 (GDM 50; AWS-ML 0). Filters: dedup_rm=32, case-study_rm=7, sub-1KB-stub_rm=49, trafilatura-upgrade=0 (all byte-identical to 16:20). Final inbox 0→172 (big=74, mid=65, small=33), 172 unique URLs, 0 intra-inbox dups. Count +1 vs 16:20 baseline 171 — decomposed: the +1 = exactly 1 genuinely-new WeChat-新智元 big file (recovery 新智元 18→19; scan New=3 but only +1 net — 2 of the 3 New were stale DB re-reports of already-captured articles, the URL-format mismatch in DB read-state marking, documented pattern). Composition 73/65/33 → 74/65/33 (+1 big) maps exactly; all other components byte-identical (curl 50, HF 5, dedup 32, case-study 7, sub-1KB 49, traf 0, all other recovery feeds flat). Reconciliation: 209 + 50 + 5 (HF) = 264 gross − 32 − 7 − 49 = 176, minus 4 already-ingested-skipped → 172 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD with organic WeChat-新智元 cascade (+1 big), NOT stale/feed-health — the +1 is decomposed exactly to the single new 新智元 big article; zero feed-death signals (all std + curl HTTP=200). URL-200=160/172 first pass (aws 23/23, mp.weixin 130/133, interconnects 6/7, oneusefulthing 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient); patient retry resolved all mp.weixin/interconnects failures — connection-level transients, DNS healthy, no escalation).
Prior session log: references/cron-run-2026-08-27-1620.md (2026-08-27 16:20, 7th run of day — FRESH-CYCLE REBUILD, byte-identical to 14:10 baseline — 0→171 (big=73, mid=65, small=33): start inbox=0 (drained 15:20 wiki-inbox-scan-v2 + 15:43 wechat-inbox-pipeline, both post-14:10 with 0 rss ingest). 7 std scans: all 7 New=0 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): all 11 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=208 written, 107 skipped — byte-identical to 14:10 (std 47: AWS-CB 9, AWS-ML 14, CrewAI 9, Interconnects 8, Netflix 6, OUT 1 + WeChat 161: 量子位 18, 夕小瑶 18, 字节 15, 小米 17, 新智元 18, 机器之心 13, 美团 17, 腾讯 10, 阿里云开发者 15, 阿里技术 9, PaperWeekly 11); watchdog=0 written (GDM short-circuit, all scans New=0); curl-recovery=50 (GDM 50; AWS-ML 0). Filters: dedup_rm=32, case-study_rm=7, sub-1KB-stub_rm=49, trafilatura-upgrade=0. Final inbox 0→171 (big=73, mid=65, small=33), 171 unique URLs, 0 intra-inbox dups. Count 0 vs 14:10 baseline 171 — exactly byte-identical on EVERY component (recovery 208, curl 50, HF 5, dedup 32, case-study 7, sub-1KB 49, traf 0, final 171, composition 73/65/33, URL-200 163/171); all scans New=0 → no organic content; watchdog short-circuited confirms; no ingestion since 14:10 (both drains 0 rss ingest); zero feed-death signals. Reconciliation: 208 + 50 + 5 (HF) = 263 gross − 32 − 7 − 49 = 175, minus 4 already-ingested-skipped → 171 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD, byte-identical deterministic steady state, NOT stale/feed-health — pure determinism, no delta on any component, no escalation. URL-200=163/171 (aws 23/23, mp.weixin 132/132, interconnects 7/7, oneusefulthing 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy).
Prior session log: references/cron-run-2026-08-27-1410.md (2026-08-27 14:10, 6th run of day — FRESH-CYCLE REBUILD, byte-identical to 12:03 baseline #10 — 0→171 (big=73, mid=65, small=33): start inbox=0 (drained 13:11 wiki-inbox-scan-v2 + 13:31 wechat-inbox-pipeline, both post-12:03 with 0 rss ingest). 7 std scans: all 7 New=0 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): all 11 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=208 written, 107 skipped — byte-identical to 12:03 (std 47: AWS-CB 9, AWS-ML 14, CrewAI 9, Interconnects 8, Netflix 6, OUT 1 + WeChat 161: 量子位 18, 夕小瑶 18, 字节 15, 小米 17, 新智元 18, 机器之心 13, 美团 17, 腾讯 10, 阿里云开发者 15, 阿里技术 9, PaperWeekly 11); watchdog=0 written (GDM short-circuit, all scans New=0); curl-recovery=50 (GDM 50; AWS-ML 0). Filters: dedup_rm=32, case-study_rm=7, sub-1KB-stub_rm=49, trafilatura-upgrade=0. Final inbox 0→171 (big=73, mid=65, small=33), 171 unique URLs, 0 intra-inbox dups. Count 0 vs 12:03 baseline 171 — exactly byte-identical on EVERY component (recovery 208, curl 50, HF 5, dedup 32, case-study 7, sub-1KB 49, traf 0, final 171, composition 73/65/33, URL-200 163/171); all scans New=0 → no organic content; watchdog short-circuited confirms; no ingestion since 12:03 (13:11 + 13:31 drains both 0 rss ingest); zero feed-death signals. Reconciliation: 208 + 50 + 5 (HF) = 263 gross − 32 − 7 − 49 = 175, minus 4 already-ingested-skipped → 171 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD, byte-identical deterministic steady state, NOT stale/feed-health — pure determinism, no delta on any component, no escalation. URL-200=163/171 (aws 23/23, mp.weixin 132/132, interconnects 7/7, oneusefulthing 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy).
Prior session log: references/cron-run-2026-08-27-1203.md (2026-08-27 12:03, 5th run of day — FRESH-CYCLE REBUILD, byte-identical to 10:00 baseline #9 — 0→171 (big=73, mid=65, small=33): start inbox=0 (drained by 10:11 + 11:20 wechat-inbox-pipeline runs, both with 0 rss ingest). 7 std scans: all 7 New=0 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): all 11 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=208 written, 107 skipped — byte-identical to 10:00 (std 47: AWS-CB 9, AWS-ML 14, CrewAI 9, Interconnects 8, Netflix 6, OUT 1 + WeChat 161: 量子位 18, 夕小瑶 18, 字节 15, 小米 17, 新智元 18, 机器之心 13, 美团 17, 腾讯 10, 阿里云开发者 15, 阿里技术 9, PaperWeekly 11); watchdog=0 written (GDM short-circuit, all scans New=0); curl-recovery=50 (GDM 50; AWS-ML 0). Filters: dedup_rm=32, case-study_rm=7, sub-1KB-stub_rm=49, trafilatura-upgrade=0. Final inbox 0→171 (big=73, mid=65, small=33), 171 unique URLs, 0 intra-inbox dups. Count 0 vs 10:00 baseline 171 — exactly byte-identical on EVERY component (recovery 208, curl 50, HF 5, dedup 32, case-study 7, sub-1KB 49, traf 0, final 171, composition 73/65/33, URL-200 163/171); all scans New=0 → no organic content; watchdog short-circuited confirms; no ingestion since 10:00 (both drains 0 ingest); zero feed-death signals. Reconciliation: 208 + 50 + 5 (HF) = 263 gross − 32 − 7 − 49 = 175, minus 4 already-ingested-skipped → 171 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD, byte-identical deterministic steady state, NOT stale/feed-health — pure determinism, no delta on any component, no escalation. URL-200=163/171 (aws 23/23, mp.weixin 132/132, interconnects 7/7, oneusefulthing 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy).
Prior session log: references/cron-run-2026-08-27-0955.md (2026-08-27 09:55, 4th run of day — FRESH-CYCLE REBUILD with ingestion-driven reverse shift (−2 WeChat-机器之心) — 0→171 (big=73, mid=65, small=33): start inbox=0 (drained ~08:56 wiki-inbox-scan-v2 + ~09:10 wechat-inbox-pipeline; 09:08 wechat line confirms "rss 0 (无新 rebuild)"). 7 std scans: all 7 New=0 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): all 11 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=208 written, 107 skipped — std 47 byte-identical (AWS-CB 9, AWS-ML 14, CrewAI 9, Interconnects 8, Netflix 6, OUT 1) + WeChat 161 (量子位 18, 夕小瑶 18, 字节 15, 小米 17, 新智元 18, 机器之心 13 [15→13 −2 ingestion], 美团 17, 腾讯 10, 阿里云开发者 15, 阿里技术 9, PaperWeekly 11); watchdog=0 written (GDM short-circuit, all scans New=0); curl-recovery=50 (GDM 50; AWS-ML 0). Filters: dedup_rm=32, case-study_rm=7, sub-1KB-stub_rm=49, trafilatura-upgrade=0 (all byte-identical to 05:28). Final inbox 0→171 (big=73, mid=65, small=33), 171 unique URLs, 0 intra-inbox dups. Count −2 vs 05:28 baseline 173 — decomposed: the −2 = exactly 2 WeChat-机器之心 ingests (shopify-ceo考虑禁用claude-code 4937B small + 突破遥操瓶颈全新无本体数据世界动作模型noe-0 14952B big, both verified feed_name: WeChat-机器之心 in raw/articles), recovery 机器之心 15→13 reverse shift, composition 74/65/34 → 73/65/33 (−1 big −1 small maps exactly); all other std + WeChat feeds byte-identical. Reconciliation: 208 + 50 + 5 (HF) = 263 gross − 32 − 7 − 49 = 175, minus 4 already-ingested-skipped → 171 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD with ingestion-driven reverse shift (−2 WeChat-机器之心), NOT stale/feed-health — the −2 is decomposed exactly to the 2 机器之心 ingests; zero feed-death signals. URL-200=163/171 (aws 23/23, mp.weixin 132/132, interconnects 7/7, oneusefulthing 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy).
Prior session log: references/cron-run-2026-08-27-0528.md (2026-08-27 05:28, 3rd run of day — FRESH-CYCLE REBUILD, byte-identical to 03:20 baseline — 0→173 (big=74, mid=65, small=34): start inbox=0 (drained 04:29 wiki-inbox-scan-v2 + 05:27 wechat-inbox-pipeline, both post-03:20 with 0 RSS ingest; the 05:14 ingest RSI test-time was WeChat-大模型智能, not rss-inbox). 7 std scans: all 7 New=0 (no timeouts, GDM clean). 11 WeChat scans: all 11 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=210 written, 105 skipped — byte-identical to 03:20 (std 47: AWS-CB 9, AWS-ML 14, CrewAI 9, Interconnects 8, Netflix 6, OUT 1 + WeChat 163: 量子位 18, 夕小瑶 18, 字节 15, 小米 17, 新智元 18, 机器之心 15, 美团 17, 腾讯 10, 阿里云开发者 15, 阿里技术 9, PaperWeekly 11); watchdog=0 written (GDM short-circuit, all scans New=0; AWS-ML New=0 so the 03:20 stale-re-report watchdog=1 absent); curl-recovery=50 (GDM 50; AWS-ML 0). Filters: dedup_rm=32, case-study_rm=7, sub-1KB-stub_rm=49, trafilatura-upgrade=0. Final inbox 0→173 (big=74, mid=65, small=34), 173 unique URLs, 0 intra-inbox dups. Count 0 vs 03:20 baseline 173 — exactly byte-identical on EVERY component (recovery 210, curl 50, HF 5, dedup 32, case-study 7, sub-1KB 49, traf 0, final 173, URL-200 165/173); all scans New=0 → no organic content; watchdog short-circuited confirms; zero feed-death signals. Reconciliation: 210 + 50 + 5 (HF) = 265 gross − 32 − 7 − 49 = 177, minus 4 already-ingested-skipped → 173 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD, byte-identical deterministic steady state, NOT stale/feed-health — pure determinism, no delta on any component, no escalation. URL-200=165/173 (aws 23/23, mp.weixin 134/134, interconnects 7/7, OUT 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy).
Prior session log: references/cron-run-2026-08-27-0320.md (2026-08-27 03:20, 2nd run of day — FRESH-CYCLE REBUILD with ingestion-driven reverse shift (−5 big) — 0→173 (big=74, mid=65, small=34): start inbox=0 (drained 02:25 wiki-inbox-scan-v2 + 03:20 wechat-inbox-pipeline; the 02:xx wiki-inbox-scan ingested 6 articles from the 01:11 rebuild's 178-file inbox → 5 rss + 1 newsletter). 7 std scans: 6 New=0; AWS-ML New=1 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): all 11 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20. Writers: recovery=210 written, 105 skipped — std 47 (AWS-CB 9, AWS-ML 14 [16→14 −2 ingestion], CrewAI 9, Interconnects 8, Netflix 6, OUT 1) + WeChat 163 (量子位 18, 夕小瑶 18, 字节 15 [16→15 −1], 小米 17, 新智元 18, 机器之心 15, 美团 17, 腾讯 10 [11→10 −1], 阿里云开发者 15, 阿里技术 9, PaperWeekly 11 [12→11 −1]); watchdog=1 written (AWS-ML, stale re-report dup → 0 net); curl-recovery=50 (GDM 50; AWS-ML 0 — already in inbox from recovery's 14). Filters: dedup_rm=32, case-study_rm=7, sub-1KB-stub_rm=49, trafilatura-upgrade=0 (all byte-identical to 01:11). Final inbox 0→173 (big=74, mid=65, small=34), 173 unique URLs, 0 intra-inbox dups. Count −5 vs 01:11 baseline 178 — decomposed: the −5 = exactly the 5 rss ingests by the 02:xx wiki-inbox-scan, all mapped to recovery per-feed reverse shifts (verified feed_name in raw/articles): AWS-ML −2 (Natera 语音 Agent 30451B + SFT 数据准备进阶 19940B), WeChat-字节 −1 (提示词注入防护 15375B), WeChat-腾讯 −1 (自进化飞轮 62775B), WeChat-PaperWeekly −1 (AutoResearch 四循环 14589B). All 5 ingested are BIG → composition big 79→74 (−5 all big), mid/small flat. AWS-ML scan New=1 + watchdog=1 written = stale re-report / dup (0 net), recovery AWS-ML exactly 14 = 16−2, NO cascade. All other std components + WeChat feeds byte-identical. Reconciliation: 210 + 50 + 5 (HF) = 265 gross − 32 − 7 − 49 = 177, minus 4 already-ingested-skipped → 173 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD with ingestion-driven reverse shift (−5), NOT stale/feed-health — the −5 is decomposed exactly to the 5 rss ingests; zero feed-death signals (all 7 std + curl HTTP=200). URL-200=165/173 (aws 23/23, mp.weixin 134/134, interconnects 7/7, OUT 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy).
Prior session log: references/cron-run-2026-08-27-0111.md (2026-08-27 01:11, 1st run of day — FRESH-CYCLE REBUILD with organic AWS-ML cascade (+2) + ingestion-driven 量子位 reverse shift (−1) — 0→178 (big=79, mid=65, small=34): start inbox=0 (drained 00:5x wechat-inbox-pipeline + 00:23 wiki-inbox-scan-v2, both saw rss 0; prior logged rebuild 20:33 wrote inbox=175; 22:53 rss-feed-scan heartbeat touch with NO cron-status line = undocumented/aborted run, self-healed). 7 std scans: GDM New=1, AWS-ML New=6, other 5 New=0 (no timeouts, GDM clean). 11 WeChat scans (proxy + --unsafe-client): 腾讯 New=1, PaperWeekly New=2, 字节 New=1 (single TIMEOUT cleared on immediate retry — in-cycle transient), other 8 New=0; proxy clean. Problem feeds: SP 0/6, HF 5/20 (baseline). Writers: recovery=215 written, 100 skipped — std 49 (AWS-CB 9, AWS-ML 16 [+2 organic], CrewAI 9, Interconnects 8, Netflix 6, OUT 1) + WeChat 166 (量子位 18 [19→18 −1 SemaPLC ingestion], 夕小瑶 18, 字节 16, 小米 17, 新智元 18, 机器之心 15, 美团 17, 腾讯 11, 阿里云开发者 15, 阿里技术 9, PaperWeekly 12); watchdog=6 written (ACTIVE via scan New>0, dups → 0 net); curl-recovery=51 (GDM 50 + AWS-ML 1). Filters: dedup_rm=32, case-study_rm=7, sub-1KB-stub_rm=49, trafilatura-upgrade=0. Final inbox 0→178 (big=79, mid=65, small=34), 178 unique URLs, 0 intra-inbox dups. Count +3 vs 20:26 baseline 175 — decomposed: (1) AWS-ML organic +2 (scan New=6 → recovery 14→16 → +2 net new big files; watchdog ACTIVE corroborates); (2) 量子位 −1 = INGESTION-driven reverse shift — SemaPLC 验证门控 PLC 代码生成 ingested 21:07, verified feed_name: WeChat-量子位 in raw/articles, now skipped (19→18); (3) curl-recovery 53→51 + dedup 34→32 + sub-1KB 50→49 = minor hygiene shifts, no feed-health change. Reconciliation: 215 + 51 + 5 (HF) = 271 gross − 32 − 7 − 49 = 183, dedup/skip overlap → 178 clean (watchdog net 0). Classification: FRESH-CYCLE REBUILD with organic AWS-ML cascade (+2) + ingestion-driven 量子位 reverse shift (−1), NOT stale/feed-health — the +3 is decomposed exactly; all std feeds except AWS-ML + other 10 WeChat feeds byte-identical; zero feed-death signals. URL-200=169/178 (aws 24/25 [1 transient], mp.weixin 137/137, interconnects 7/7, OUT 1/1; huggingface 0/4 CN wall — documented-transient; netflix 0/4 CDN osc — documented-transient; DNS healthy).
Prior session log (2026-08-26 20:26, 8th run of day): full log in references/cron-run-2026-08-26-2026.md
Prior session log (2026-08-26 16:21, 7th run of day): full log in references/cron-run-2026-08-26-1621.md
Prior session log (2026-08-26 14:05, 6th run of day): full log in references/cron-run-2026-08-26-1405.md
Prior session log (2026-08-26 09:44, 5th run of day): full log in references/cron-run-2026-08-26-0944.md
Prior session log (2026-08-26 07:42, 4th run of day): full log in references/cron-run-2026-08-26-0742.md
Prior session log (2026-08-26 05:32, 3th run of day): full log in references/cron-run-2026-08-26-0532.md
Prior session log (2026-08-26 03:23, 2th run of day): full log in references/cron-run-2026-08-26-0323.md
Prior session log (2026-08-26 01:14, 1th run of day): full log in references/cron-run-2026-08-26-0114.md
Prior session log (2026-08-25 19:53, 7th run of day): full log in references/cron-run-2026-08-25-1953.md
Prior session log (2026-08-25 17:50, 6th run of day): full log in references/cron-run-2026-08-25-1750.md
Prior session log (2026-08-25 15:34, 5th run of day): full log in references/cron-run-2026-08-25-1534.md
Prior session log (2026-08-24 21:49, 4th run of day): full log in references/cron-run-2026-08-24-2149.md
Prior session log (2026-08-24 17:00, 3rd run of day): full log in references/cron-run-2026-08-24-1700.md
Prior session log (2026-08-24 14:57, 2nd run of day): full log in references/cron-run-2026-08-24-1457.md
Prior session log (2026-08-24 12:42, 1st run of day): full log in references/cron-run-2026-08-24-1242.md
Prior session log archive (2026-08-23, runs 1-8: 00:06/02:16/04:22/06:27/08:33/10:39/12:46/20:58): pointer blocks moved to references/skill-changelog-archive-sessions-2026-08-23-full-day.md (size mgmt — SKILL.md over 100K cap); full per-run logs in references/cron-run-2026-08-23-HHMM.md
Prior session log archive (2026-08-22 13:18 → 21:56, runs 7-11): pointer blocks moved to references/skill-changelog-archive-sessions-2026-08-22-runs7-11.md (size mgmt — SKILL.md was 107,614 bytes over 100K cap); full per-run logs in references/cron-run-2026-08-22-HHMM.md
Prior session log archive (2026-08-22 00:18 → 11:07, runs 1-6): pointer blocks moved to references/skill-changelog-archive-sessions-2026-08-22-0018-to-2026-08-22-1527.md (size mgmt — SKILL.md was 102,284/100,000 chars over cap); full per-run logs in references/cron-run-2026-08-22-HHMM.md
Prior session log archive (2026-08-21 22:08 → 2026-08-20 02:00): pointer blocks for 2026-08-21 (runs 1-8: 00:13/02:22/04:28/06:33/08:38/10:44/15:53/22:08) + 2026-08-20 (runs 1-8: 02:00/04:06/06:11/08:22/10:24/15:07/17:09/22:04) moved to references/skill-changelog-archive-sessions-2026-08-21-to-2026-08-20.md (size mgmt — SKILL.md was 121,811/100,000 chars over cap, archive required before 11:07 pointer update); full per-run logs in references/cron-run-YYYY-MM-DD-HHMM.md
Prior session log archive (2026-08-19, runs 1-8: 02:10/04:13/06:21/09:14/11:33/13:39/15:47/17:57): pointer blocks moved to references/skill-changelog-archive-sessions-2026-08-19-full-day.md (size mgmt — SKILL.md was 103,583/100,000 chars over cap, archive required before next pointer update); full per-run logs in references/cron-run-2026-08-19-HHMM.md
Prior session log archive (2026-08-18 09:27 → 2026-08-18 23:53, rebuilds #31-#35): pointer blocks moved to references/skill-changelog-archive-sessions-2026-08-18-0927-to-2026-08-18-2353.md (size mgmt — SKILL.md over 100K cap); full per-run logs in references/cron-run-2026-08-18-HHMM.md.
Prior session log archive (2026-08-17 17:37 → 2026-08-18 07:20): pointer blocks for 08-17 17:37/19:59/22:03 + 08-18 02:44/04:58/07:20 moved to references/skill-changelog-archive-sessions-2026-08-17-to-2026-08-18-0720.md (size mgmt — SKILL.md was 101,176/100,000 chars over cap, archive required before 12:34 pointer update); full per-run logs in references/cron-run-YYYY-MM-DD-HHMM.md
Prior session log archive (2026-08-17 00:15 → 15:23, 7 runs): pointer blocks moved to references/skill-changelog-archive-sessions-2026-08-17-0015-to-2026-08-17-1523.md (size mgmt); full per-run logs in references/cron-run-2026-08-17-HHMM.md
Prior session log archive (2026-08-16 08:50 + 13:17): pointer blocks moved to references/skill-changelog-archive-sessions-2026-08-16-0850-to-2026-08-16-1317.md (size mgmt); full per-run logs in references/cron-run-2026-08-16-0850.md / references/cron-run-2026-08-16-1317.md
Prior session log archive (2026-08-15 runs 8-10: 18:45/21:10/23:50): pointer blocks archived to references/skill-changelog-archive-sessions-2026-08-15-1845-to-2026-08-15-2336.md (size mgmt, SKILL.md ~99.3K/100K, 2026-08-17); full per-run logs in references/cron-run-2026-08-15-1845.md / cron-run-2026-08-15-2110.md / cron-run-2026-08-15-2336.md
Prior session log archive (2026-08-15 runs 4-7: 09:39/11:49/14:13/16:27): pointer blocks archived to references/skill-changelog-archive-sessions-2026-08-15-0939-to-2026-08-15-1627.md (size mgmt, SKILL.md >100K); full per-run logs in references/cron-run-2026-08-15-0939.md / cron-run-2026-08-15-1149.md / cron-run-2026-08-15-1413.md / cron-run-2026-08-15-1627.md
Archived 2026-08-15 maintenance (size mgmt): inline blocks for 2026-08-14 17:18 / 23:51 + 2026-08-15 02:38 / 04:52 / 07:15 (runs 7-8 of 08-14 + runs 1-3 of 08-15) moved to references/skill-changelog-archive-sessions-2026-08-14-1718-to-2026-08-15-0715.md (SKILL.md was at 102,770/100,000 chars — over cap, archive required before next pointer update)
Archived 2026-08-14 maintenance (size mgmt): inline blocks for 2026-08-14 01:54 / 04:14 / 06:23 / 11:37 / 12:13 / 14:59 (runs 1-6 of 08-14) moved to references/skill-changelog-archive-sessions-2026-08-14-0154-to-2026-08-14-1459.md (SKILL.md was at 100,526/100,000 chars — over cap, archive required before next pointer update)
Archived 2026-08-14 maintenance (size mgmt): inline blocks for 2026-08-13 14:46 / 17:07 / 20:51 / 23:13 (runs 6-9 of 08-13) moved to references/skill-changelog-archive-sessions-2026-08-13-1446-to-2026-08-13-2313.md (SKILL.md was at 100,394/100,000 chars — over cap, archive required before next pointer update)
Archived 2026-08-12 maintenance (size mgmt): inline blocks for 2026-08-11 02:47 → 2026-08-11 22:52 (runs 2-10 of 08-11) + 2026-08-10 22:23 / 2026-08-11 00:37 + 2026-08-10 01:29-16:43 (runs 1-5 of 08-10) moved to references/skill-changelog-archive-sessions-2026-08-10-0129-to-2026-08-11-2252.md + references/skill-changelog-archive-sessions-2026-08-10-2223-to-2026-08-11-0037.md (SKILL.md was at 105,115/100,000 chars — over cap, archive required before pointer update)
Archived 2026-08-10 maintenance (size mgmt): inline blocks for 2026-08-09 00:38 → 23:13 (runs 1-12 of 08-09) + 2026-08-08 01:52-15:07 (runs 1-7 of 08-08) + 2026-08-07 16:56-23:36 (runs 7-10 of 08-07) + 2026-08-06 04:09-22:23 (runs 6-10 of 08-06) moved to references/skill-changelog-archive-sessions-2026-08-06-0409-to-2026-08-06-2223.md / -2026-08-07-1656-to-2026-08-07-2336.md / -2026-08-08-0152-to-2026-08-08-1507.md / -2026-08-09-0038-to-2026-08-09-2313.md (SKILL.md at cap during each; archive required before pointer update)
Archived inline blocks for rebuilds #25–#33 (2026-08-05 05:38 → 23:51) moved to references/skill-changelog-archive-sessions-2026-08-05-0538-to-2026-08-05-2351.md (2026-08-06 maintenance); see references/cron-run-YYYY-MM-DD-HHMM.md for full per-run logs.
Archived session logs (2026-08-02 → 2026-08-05 03:28, rebuilds #4–#27): inline blocks removed 2026-08-05 to stay under the 100K SKILL.md cap; one-line index in references/skill-changelog-archive-sessions-2026-08-02-to-08-05-0328.md, full per-run logs in references/cron-run-YYYY-MM-DD-HHMM.md. Rebuild chain: #4 (08-02 04:26) → #11 (08-02 21:18) → #12 (08-02 23:24) → #13/#14 (08-03) → #15/#16/#17 (08-03) → #18 (08-03 18:46) → #19 (08-03 23:21) → #20–#23 (08-04) → #24/#25 (08-04) → #26/#27 (08-05).
v5.9.23.30 → v5.9.23.44 archived: Archived in session reference files. See individual cron-run-YYYY-MM-DD-HHMM.md files.
Problem (observed 2026-07-03 05:57): The SKILL.md on disk was found at 1,806 bytes — truncated to only the frontmatter + "RSS 策略" header + placeholder text. The full body (103KB) was lost. Root cause: a prior skill_manage(action='edit') call that passed only the first few lines instead of the complete file, overwriting the full content with a stub.
Fix (applied in v5.9.23.50): Restored from the wiki git HEAD (v5.9.23.41) and updated the session pointer + new sections. Older verbose changelog blocks (v5.9.23.x SKILL.md update narrative blocks) were removed to keep the file under the 100K char limit — they are preserved in individual session reference files (references/cron-run-*.md).
Prevention:
edit call on this skill, run skill_view('rss-to-wiki-pipeline') and verify the full body is present. The >50KB byte threshold is STALE (v5.9.23.93): the file was deliberately slimmed in the v5.9.23.50 cleanup; healthy size is now ~42KB. Verify by grepping key sections (## Stable Operational Patterns, ## Latest Session, ## RSS 策略) instead of byte size. Use anchored grep -n '^## <Section>' for the uniqueness check — unanchored grep -c '## <Section>' returns 2 for headings that also appear in prose (e.g. ## RSS 策略 is referenced in the corruption-incident section), which triggers a false duplicate-header alarm (verified 2026-08-03). DO NOT byte-compare to git HEAD — HEAD (36.8KB) is legitimately smaller than the on-disk file (41.9KB) because session-log pointers accumulate in the working copy; a byte-based false alarm triggers a destructive restore that regresses the file.edit, ALWAYS pass a COMPLETE SKILL.md, not a partial reconstruction.~/wiki/skills/) is the recovery source — git show HEAD:skills/wiki/rss-to-wiki-pipeline/SKILL.md has the canonical copy.| 保留 | 源 | 文章类型 | 知识价值 |
|---|---|---|---|
| ✅ | Hugging Face Blog | Post-Training / Agent / Eval / 推理 | ⭐⭐⭐ — 每篇直接命中知识库核心（vLLM/RL/Agent失败模式/DeepSeek-V4）。RSS URL: https://huggingface.co/blog/feed.xml（非 /blog/rss.xml）。feed 无 content:encoded，CN 网络下 curl 返回 0 字节（JS SPA 完全被墙），必须 TinyFish 或 VPN。 |
| ✅ | Google DeepMind Blog | AI 前沿研究发布（Alpha/RL/Agent） | ⭐⭐⭐⭐ — 替代 NVIDIA，有独立知识深度 |
| ✅ | Stochastic Parrot | AI 哲学/社会/认知深度长文 | ⭐⭐⭐ — 有独立知识体系 |
| ✅ | One Useful Thing | AI 与工作/社会交叉分析（Ethan Mollick） | ⭐⭐ — 教育性应用分析 |
| ✅ | CrewAI Blog | 多 Agent 框架官方博客 | ⭐⭐ — 有技术深度，但偏产品 |
| ✅ | Netflix Tech Blog | PERMANENT (v5.9.23.35→v5.9.23.40, 6+ crons)：Feed URL returned to life after ~2 weeks of 404s. Articles return 403 via curl (medium CDN) but recovery captures full 10-36KB content from feed entries. Include in scan chains. | |
| ✅ | AWS China Blog (中文) | AWS 中国区原创技术文章 | ⭐⭐ — 中文原创，Agent 实践 |
| ✅ | AWS China ML | AWS ML 技术文章 | ⭐⭐ — 英文，但有 Agent/ML 内容 |
| 关闭 | 源 | 关闭原因 |
|---|---|---|
| ❌ | Import AI | newsletter digest，每期 3-5 话题都很浅，无法独立入库 |
| ❌ | Latent Space | AINews digest，同上 |
| ❌ | Last Week in AI | AI news digest |
| ❌ | Cloudflare AI Blog | 公司产品公告为主 |
| ❌ | AWS 英文分类 blogs (12) | 产品发布/文档为主，知识密度低 |
| ❌ | Stanford SAIL | summary only，极少更新 |
| ❌ | NVIDIA Developer Blog | 永久下架（2026-05-08）：RSS 推送的 2024-2025 年文章 URL 全部 404（原站已删文），feed 本身无新文章价值。58 篇已入库旧文已清理。 |
RSS 文章只有满足以下条件才值得评分入库：
如果一篇文章不满足以上条件，就留在 raw/rss-inbox/ 中。不强行入库。
Q: 这篇文章我读完能提炼出什么？
→ 能提炼出一个实体、一个概念、或一个工程原则 → 入库
→ 只能提炼出"哦有这么件事" → 不入库
Q: 这篇文章在我知识库里能链接到哪里？
→ 有 2 个以上现有页面可以交叉链接 → 入库
→ 孤立存在，没有链接 → 不入库
| 特征 | News Digest | 独立长文 |
|---|---|---|
| 内容结构 | 3-5个话题拼凑，每个一段摘要 | 单一主题，有完整论据 |
| 入库后得到 | 空壳（只有归类+原文链接） | 有知识提炼的实体/概念页 |
| 交叉链接 | 无法建立有意义的链接 | 可连接2+现有页面 |
| 典型来源 | Import AI, Latent Space AINews | Google DeepMind Blog, Stochastic Parrot |
关键教训： 把整期 newsletter 当成一篇文章入库，得到的 entity 只有归类+原文链接，和 bookmark 没区别。这和知识库"沉淀可复用知识"的初衷背道而驰。不要入库 newsletter digest。
重要教训：不要把所有步骤放在同一个 cron 里。 合并 RSS + 微信 + Newsletter 会导致单步超时中断整个任务。应将抓取步骤拆为独立子 cron，汇总报告单独一个 cron。
4 个独立 cron 任务，每 8h 运行，互不阻塞：
rss-feed-scan → raw/rss-inbox/ (7 good feeds)wechat-mp-scan → raw/wechat-inbox/newsletter-link-extract → candidates.mdwiki-inbox-scan-v2 → 汇总 + 邮件报告cron 运行在非交互式 shell，不加载 .zshrc。注入方式：
source ~/.wiki-cron.env
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY
所有脚本位于 ~/.hermes/skills/wiki/rss-to-wiki-pipeline/scripts/ 下：
rss-inbox-recovery.py — 主路径rss-inbox-watchdog.py — 辅助路径（当 recovery 漏抓时）rss-inbox-curl-recovery.py — GDM 兜底；也补充其他 feeds 的被 auto-mark-read 文章rss-inbox-trafilatura-upgrade.py — 对 sub-2KB 文件尝试 trafilatura 升级inbox-case-study-filter.py — 删除 vendor case-study PR 文章rss-inbox-sub-1kb-filter.py — 删除 sub-1KB trafilatura stubfetch-problem-feeds.py — 抓取 Substack + HuggingFace（Go HTTP timeout 的 feeds）cron-log-write.sh — 将结构化 log 行写入 cron-status.logrss-source-reaper.py — 清理失效 RSS sourcesImportant: Always unset proxy variables before running blogwatcher-cli — the Go HTTP client fails with proxy authorization errors in the cron environment.
# Step 1: Touch heartbeat
python3 ~/wiki/scripts/cron-heartbeat.py touch rss-feed-scan
# Step 2: Scan good feeds (blogwatcher-cli, individual, 20s per feed timeout)
# IMPORTANT: use Python subprocess.run(timeout=20) wrapper — shell `timeout` command NOT available in cron env
# ⚠️ subprocess.run INHERITS proxy env vars — blogwatcher-cli's Go HTTP client fails with
# "not authorized by the client" when proxies are set. Always pass env with proxies removed.
# Do NOT use env={} (clears PATH+HOME → blogwatcher-cli: not found + $HOME not defined).
# Netflix Tech Blog reactivated — feed 200, articles 403 via curl, recovery captures content
/usr/bin/python3 -c "
import subprocess, os
env = os.environ.copy()
for k in ['http_proxy', 'https_proxy', 'HTTP_PROXY', 'HTTPS_PROXY']:
env.pop(k, None)
for feed in ['Google DeepMind Blog', 'One Useful Thing', 'Netflix Tech Blog', 'CrewAI Blog', 'Interconnects', 'AWS China Blog', 'AWS China ML']:
print(f'=== {feed} ===', flush=True)
try:
r = subprocess.run(['blogwatcher-cli', 'scan', feed], capture_output=True, text=True, timeout=20, env=env)
print(r.stdout[-500:] if r.stdout else '(no stdout)', flush=True)
if r.stderr: print('ERR:', r.stderr[-300:], flush=True)
except subprocess.TimeoutExpired:
print(f'TIMEOUT (20s) on {feed}', flush=True)
"
# Step 3: Scan problem feeds (Python fallback)
/usr/bin/python3 ~/.hermes/skills/wiki/rss-to-wiki-pipeline/scripts/fetch-problem-feeds.py
# Step 4: Write new articles to inbox (recovery is main path, watchdog is secondary)
# ⚠️ recovery.py can exceed default terminal timeout (120s) when SSL handshake
# delays accumulate across multiple feeds. Always use timeout ≥300s when
# calling via terminal(). If first attempt times out, retry with longer timeout.
/usr/bin/python3 ~/.hermes/skills/wiki/rss-to-wiki-pipeline/scripts/rss-inbox-recovery.py 2>&1 | tail -20
/usr/bin/python3 ~/.hermes/skills/wiki/rss-to-wiki-pipeline/scripts/rss-inbox-watchdog.py 2>&1 | tail -5
/usr/bin/python3 ~/.hermes/skills/wiki/rss-to-wiki-pipeline/scripts/rss-inbox-curl-recovery.py 2>&1 | tail -30
# Step 5: Remove duplicates already ingested to raw/articles/
# ⚠️ CRITICAL: Use FULL URL matching, NOT slug substring matching.
# ⚠️ TIMEOUT RISK: raw/articles/ has grown to ~3000 files. The glob+read+regex loop
# can exceed terminal's default 30s timeout. Always use terminal(..., timeout=120)
# or higher when calling the inline dedup script. If first attempt times out, retry
# with 120s — the script will complete.
# ⚠️ QUOTE NORMALIZATION (2026-08-16, rebuild 13:17): raw/articles stores ~1/3 of
# source_url values QUOTED ("https://...") while inbox files write unquoted URLs.
# The pre-fix script compared exact strings → 42 already-ingested articles re-entered
# the inbox EVERY cycle (recovery's own skip logic has the same blind spot). Strip
# surrounding quotes on BOTH sides before comparing.
/usr/bin/python3 -c "
import os, re
from pathlib import Path
wiki_root = Path.home() / 'wiki'
inbox_dir = wiki_root / 'raw/rss-inbox'
articles_dir = wiki_root / 'raw/articles'
def norm_url(u):
return u.strip().strip('\"').strip(\"'\").rstrip('/')
# Build set of FULL normalized URLs from raw/articles/
ingested_urls = set()
for f in articles_dir.glob('*.md'):
try:
content = f.read_text()
m = re.search(r'^(?:source_)?url:\\s*(.+)$', content, re.MULTILINE)
if m:
ingested_urls.add(norm_url(m.group(1)))
except: pass
# Remove inbox files whose FULL URL matches an ingested URL
removed = 0
for f in inbox_dir.glob('*.md'):
try:
content = f.read_text()
m = re.search(r'^(?:source_)?url:\\s*(.+)$', content, re.MULTILINE)
if m:
url = norm_url(m.group(1))
if url in ingested_urls:
f.unlink()
removed += 1
except: pass
print(f'Removed {removed} duplicates from inbox')
"
# Step 5b: Auto-filter vendor case-study PRs
/usr/bin/python3 ~/.hermes/skills/wiki/rss-to-wiki-pipeline/scripts/inbox-case-study-filter.py --threshold 1 --max-size 8192
# Step 5c: Remove sub-1KB trafilatura stubs
/usr/bin/python3 ~/.hermes/skills/wiki/rss-to-wiki-pipeline/scripts/rss-inbox-sub-1kb-filter.py
# Step 5d: Try trafilatura upgrade on remaining sub-2KB files
/usr/bin/python3 ~/.hermes/skills/wiki/rss-to-wiki-pipeline/scripts/rss-inbox-trafilatura-upgrade.py
# Step 6: URL health check (after all filter steps) — with per-domain breakdown
# IMPORTANT: Use 180s terminal timeout. Individual checks with --connect-timeout 2 avoid
# the DNS-latency wall that batching can hit. See "URL health check" pitfall.
/usr/bin/python3 -c "
import os, re, subprocess
from collections import defaultdict
from pathlib import Path
from urllib.parse import urlparse
inbox_dir = Path.home() / 'wiki' / 'raw' / 'rss-inbox'
domain_stats = defaultdict(lambda: {'ok': 0, 'fail': 0})
checked = 0
healthy = 0
for f in sorted(inbox_dir.glob('*.md')):
content = f.read_text()
m = re.search(r'^(?:source_)?url:\\\\s*(https?://[^\\s]+)', content, re.MULTILINE)
if not m:
continue
url = m.group(1).strip()
domain = urlparse(url).netloc
try:
r = subprocess.run(['curl', '-sL', '-o', '/dev/null', '-w', '%{http_code}', '--connect-timeout', '2', '--max-time', '5', url], capture_output=True, text=True, timeout=7)
code = r.stdout.strip()
if code in ('200', '301', '302', '307'):
domain_stats[domain]['ok'] += 1
healthy += 1
else:
domain_stats[domain]['fail'] += 1
checked += 1
except subprocess.TimeoutExpired:
domain_stats[domain]['fail'] += 1
checked += 1
print(f'URL-200={healthy}/{checked}')
for d in sorted(domain_stats.keys()):
s = domain_stats[d]
print(f' {d}: {s[\\\"ok\\\"]}/{s[\\\"ok\\\"]+s[\\\"fail\\\"]}')
"
# Step 7: Write to cron-status.log via wrapper (ABSOLUTE path required)
# IMPORTANT: assemble LINE via shell single-quoted string in terminal(), NOT execute_code
# (execute_code is blocked in cron mode)
/usr/bin/python3 -c "import os; from pathlib import Path; d=Path.home()/'wiki'/'raw'/'rss-inbox'; files=list(d.glob('*.md')); big=sum(1 for f in files if f.stat().st_size>=10240); mid=sum(1 for f in files if 5120<=f.stat().st_size<10240); small=sum(1 for f in files if f.stat().st_size<5120); print(f'big={big} mid={mid} small={small} total={len(files)}')"
# Then assemble LINE variable and write via:
# bash ~/.hermes/skills/wiki/rss-to-wiki-pipeline/scripts/cron-log-write.sh "$LINE"
每行 rss-feed-scan 记录使用以下 schema（保持可解析、可 diff）：
[YYYY-MM-DD HH:MM:SS] rss-feed-scan: scanned=<N> (<feed list summary>) + problem_feeds(<SP>+<HF>) | recovery=<R> written, <S> skipped | watchdog=<W> written (active when any feed unread>0; else 0 = short-circuit) | curl-recovery=<C> written (de-facto main path when urllib SSL times out; 0 = curl-fallback didn't run / recovered nothing / OR all URLs already in inbox from recovery — observed 2026-08-01 16:51: "32 URLs already in rss-inbox/" dedup means curl-recovery adds nothing when recovery.py ran first) | dedup_rm=<D> | case-study_rm=<CS> | sub-1KB-stub_rm=<S1> | trafilatura-upgrade=<TU> | inbox=<T> files (>=10KB=<B>, mid=<M>, small=<S2>) | URL-200=<H>/<T>
这些模式在多次独立 cron 中确认复现，未来 session 看到同样的指标组合可直接信任：
| 模式 | 触发条件 | 预期结果 | 验证 |
|---|---|---|---|
| GDM dead-end N=60 stable | GDM inbox=0 after filter chain | Skip GDM in cron health assessment. Two paths: (a) HTTP=0 → 0 GDM; (b) HTTP=200 with 50-60 stubs → sub-1KB filter removes → 0 in inbox. Don't equate "0 GDM" with "feed dead". Variant can REPEAT consecutively (2026-08-02 REFINES the 08-01 16:51 'alternation' claim): do NOT expect alternation between (a) curl-recovery=0/sub-1KB-stub_rm=0 and (b) curl-recovery=53/sub-1KB-stub_rm=53 — either variant can run consecutively (observed 4 straight variant-b cycles 08-02 04:26→10:39; 16:51 was variant-a, isolated). Both converge to 0 GDM in inbox with the same byte-identical 28-file signature. Any curl-recovery/sub-1KB count = GDM variant marker, NOT an anomaly or feed-health change; never read it as feed health. | 60+ independent crons |
| Inbox shape = 31-39 normal (post-Netflix baseline, v5.9.23.38 REFINED); fully-drained steady state ~28-32 (v5.9.23.62 ADDED); extreme deep-drain <20 (v5.9.23.64 ADDED); composition range broadened 2026-07-29 (v5.9.23.81 REFINED) | 7 good feeds after backlog sweep | 31-39 files = new normal when any feed still has backlog beyond CrewAI. Composition: 25-31 >=10KB + 6-8 mid + 0 small. Fully-drained steady state (v5.9.23.62+): inbox=28-32 (22-24 >=10KB + 6-8 mid + 0 small), below 31-39 range when all backlog feeds exhausted. If inbox drops below 25: check whether OUT milestone is absent (expected, causes additional drop) vs. feed health issue. Extreme deep-drain (v5.9.23.64, range broadened v5.9.23.81, dual-trigger confirmed v5.9.23.82): Two trigger paths: (a) recovery net contribution ~0 for sustained cycles (all feeds New=0, all recovered articles filtered by downstream filters, OUT milestone ingested) — inbox bottoms at 12-15; (b) non-zero recovery but high filter consumption — observed 2026-07-30: recovery=23 with 4 case-study + 54 sub-1KB removed → inbox=19. In both cases, variable composition applies: previously observed at 4-6 big + 6-8 mid (v5.9.23.64), but also observed at 10 big + 5 mid (2026-07-29) when AWS big-files dominate the residual set. The composition is NOT fixed — it depends on which feed's articles survive the filter chain. What matters is the total count <20. This is NOT a feed health signal — it means ALL backlogs are fully exhausted and even prior-cycle residuals are being consumed. URL health check at extreme-drain typically shows 12-15/12-15 reachable. v5.9.23.82 variant added (2026-07-30): extreme deep-drain can ALSO occur with non-zero recovery (e.g. 23) when filters consume the majority of incoming articles. Recovery=23 with 4 case-study + 54 sub-1KB removed → net inbox=19 (14 big + 5 mid). The trigger is filter chain efficiency rather than recovery floor. Was previously recorded as a separate mode (v5.9.23.65), but inbox bounced back from 24→29 in the next cycle, confirming sub-28 values are oscillatory dips within the fully-drained steady state, not a permanent downward shift. The inbox fluctuates within 28-32 nominal with occasional dips to 24-25 when recovery net is near-zero and mid-size files are consumed faster than big files. Treat intermediate-range inbox as "fully-drained steady state with transient dip", not a separate trajectory. during AWS-CB/ML partial-drain (recovery=23-28), inbox=24 (18 big + 5 mid + 1 small) HELD STEADY across 2 consecutive cycles (2026-08-01 06:22 + 08:26, identical composition) — a persistent partial-drain shape, NOT a transient dip. Distinguish by recovery value: recovery=28 with inbox=24 = stable partial-drain; recovery≈6-10 with inbox=24 = oscillatory dip within fully-drained state. |
| 48+ crons; oscillation confirmed v5.9.23.67; broadened composition v5.9.23.81; filter-driven extreme-drain variant v5.9.23.82; confirmed recurring v5.9.23.85 (4x same day 2026-07-30); AWS-only transient URL pattern (1/11) also holds at extreme-drain, but can be absent in some cycles (all 11/11 reachable 2026-07-30 09:41); AWS transient severity varies within same day — narrow (1/11), broad (7/11), absent — all resolve within seconds on retry; 5th occurrence same day (v5.9.23.86 hindsight review) |
| Recovery 38-46 normal (post-Netflix baseline, v5.9.23.38 REFINED); CrewAI-only floor ~6 (v5.9.23.58 ADDED) | Recovery main path, Netflix reactivated | 38-46 written per sweep when OUT milestone present. Variables: CrewAI 8-11 + AWS-CN-ML 11-15 + Interconnects 5-6 + AWS-CB 4 + Netflix 9 + OUT 1. When OUT milestone is absent (ingested), recovery drops to 30-35 — this was a TRANSITIONAL range (v5.9.23.50-54) driven by residual AWS-CB-ML backlog. After full backlog exhaustion (~v5.9.23.62), the true fully-drained steady state is recovery=6-10 (CrewAI floor 6 + AWS-CB trickle 2-4). Intermediate partial-drain recovery ~23-28 (v5.9.23.82 ADDED, range broadened v5.9.23.96): when AWS-CB/ML backlogs are partially drained but not exhausted, recovery can settle at 23-28. Observed 2026-07-30: recovery=23 (AWS-CB 5, AWS-ML 6, CrewAI 6, Interconnects 1, Netflix 4, OUT 1) — AWS-CB/ML contributions (11 combined). Confirmed 2026-08-01 (2 consecutive cycles): recovery=28 ×2 (AWS-CB 6, AWS-ML 10, CrewAI 6, Interconnects 1, Netflix 4, OUT 1) — AWS-CB/ML combined = 16. AWS-CB/ML contributions (11-16 combined) elevate recovery above the 6-10 floor but below the 30-35 transitional range. This is a distinct intermediate state between "transitional" and "fully-drained". Extreme floor: when all feeds have New=0 in blogwatcher scan AND the only auto-marked-read articles are CrewAI case-study PRs, recovery can drop to ~6 (all removed by case-study filter) — net contribution ~0. This is NOT a feed health signal; it means the backlog is fully drained across all feeds. Fresh-cycle rebuild (v5.9.23.98 ADDED, confirmed 2026-08-01 12:36; DOUBLE-confirmed v5.9.23.99 2026-08-01 14:38): when wiki-inbox-scan-v2 drains the inbox BETWEEN cron runs (start inbox=0), the next rss-feed-scan is a rebuild cycle: recovery=32 (AWS-CB 6, AWS-ML 10, CrewAI 9, Interconnects 2, Netflix 4, OUT 1) rebuilds inbox 0 → 28 (20 big + 7 mid + 1 small) back into the fully-drained steady-state band 28-32. Distinguish from partial-drain by START STATE: start inbox=0 + recovery≥28 = fresh rebuild; start inbox=24 held + recovery=28 = partial-drain. CrewAI recovery varies 6-9 by case-study PR backlog (4 filtered → net 5 in inbox); AWS-CB 6 + AWS-ML 10 held stable at 16 combined across all 5 runs on 2026-08-01. Determinism insight (v5.9.23.99, SEPTUPLE-confirmed 2026-08-02 10:39; EXTENDED to 14 consecutive rebuilds 2026-08-03 03:40): SEVEN consecutive fresh-cycle rebuilds spanning 2+ days (08-01 12:36/14:38/16:51 + 08-02 04:26/06:30/08:37/10:39) produced BYTE-IDENTICAL signatures (same per-feed recovery splits, case-study_rm=4, sub-1KB-stub_rm=53, curl-recovery=53, inbox 0→28 with identical composition). Cross-day determinism is now ESTABLISHED — not day-scoped. Signature shift held (2026-08-03, rebuilds #11-#14): the deterministic signature moved 0→28 → 0→29 (big=20, mid=8, small=1) at rebuild #11 (08-02 21:18, organic +1 latest-open-artifacts-23 un-ingested) and has held 0→29 for 4 consecutive rebuilds — a persistent un-ingested organic +1 SHIFTS the signature but does NOT break determinism; expect 0→29, not 0→28, until latest-open-artifacts-23 is ingested. The rebuild numbers are deterministic given (a) drained start inbox=0 and (b) unchanged raw/articles/ ingested set — identical recovery counts across consecutive fresh cycles = expected determinism, NOT a stuck pipeline. Organic-variation carve-out (confirmed 2026-08-02 21:18, rebuild #11): the byte-identical claim holds only when no feed publishes between crons. If any feed shows New>0 in the blogwatcher scan, expect recovery +1, watchdog flips short-circuit(0)→active(1), and inbox +1 (typically +1 mid file) vs the deterministic signature — all other components stay identical (case-study_rm, sub-1KB-stub_rm, per-feed splits unchanged). When the inbox deviates by exactly this pattern, check the scan New-counts FIRST: it is organic new content, NOT a stuck pipeline or feed-health change. LIVE-confirmed in action 2026-08-03 23:21 (rebuild #19) — first full in-session observation of the cascade: Interconnects scan New=1 → recovery 33→34 (Interconnects 3→4), watchdog short-circuit(0)→active(1), inbox 0→29 → 0→30 (+1 mid introducing-our-artifacts-hub-and-adoption-dashboard.md 5137B), case-study_rm=4 + sub-1KB-stub_rm=53 + curl-recovery=53 identical to the deterministic signature. The carve-out predicted exactly what ran — treat any deviation matching this cascade as organic content, never as stuck pipeline. Generalized +N form (2026-08-04 01:34, rebuild #20 — second in-session observation): the cascade is NOT limited to +1 mid file from Interconnects. AWS-ML scan New=2 → recovery 34→36 (AWS-ML 10→12), watchdog active (2 written, 100% hit — not short-circuited), inbox 0→30 → 0→32 (big 20→22, mid 9, small 1 — +2 BIG files), filters byte-identical (case-study_rm=4, sub-1KB-stub_rm=53, curl-recovery=53, dedup_rm=0, trafilatura-upgrade=0). Rule: delta count N = the publishing feed's scan New-count; the +N files' size profile follows THAT feed (AWS-ML → big, Interconnects → mid, AWS-CB → big — 3rd confirmed trigger, rebuild #24 2026-08-04 15:17: AWS-CB New=1 → recovery 36→37, watchdog active, inbox 0→32→0→33; 4th confirmed trigger rebuild #26 2026-08-05 01:20 (AWS-ML New=1 → recovery 37→38, watchdog active, inbox 0→33→0→34); 5th confirmed trigger rebuild #36 2026-08-06 04:09 (AWS-ML New=4 → recovery 38→42 (13→17), watchdog active (4 dups), inbox 0→34→0→38 — +4 ALL-BIG, composition 24/9/1→28/9/1, largest delta yet) — AWS-ML has now triggered the cascade THREE times (rebuilds #20/#26/#36), proving triggers are feed-REPEATABLE, not one-per-feed; any feed can trigger multiple times as it publishes new articles; delta N scales with the scan New-count (1→1, 2→2, 4→4) and the size profile follows the publishing feed (AWS-ML → big). The baseline moved 0→34 → 0→38 from #36, persisting until the 4 new AWS-ML articles are ingested. Read scan New-counts FIRST and size the expected delta to the feed — a +N big-file deviation is organic content, never an anomaly. PERSISTENCE refinement (2026-08-04 03:36, rebuild #21 — third in-session observation, byte-identical to #20): the +N shift PERSISTS across subsequent rebuilds while the organic articles stay un-ingested. Rebuild #21 ran with ALL 7 scans New=0 (no new organic content; watchdog short-circuited 0/5) yet recovery STILL=36 (AWS-CB 6, AWS-ML 12, CrewAI 9, Interconnects 4, Netflix 4, OUT 1) and inbox 0→32 (big=22, mid=9, small=1). The un-ingested AWS-ML +2 (from #20's New=2: generate-autonomous-business-insights-with-ai-agent-and-mcp-, automating-customer-retention-workflows-in-amazon-quick) and Interconnects +1 latest-open-artifacts-23 (from #19's New=1) re-capture every rebuild, so the deterministic baseline itself moved: 0→29 (#11-#18) → 0→32 (#20-#23 — holds 4 consecutive rebuilds, confirmed 2026-08-04 07:45 with all scans New=0 + watchdog short-circuited yet recovery STILL=36) → 0→33 (#24-#25, holds 2 consecutive rebuilds — the AWS-CB +1 from #24 joined the limbo set; byte-identical composition 23/9/1 both cycles, confirmed 2026-08-04 17:25 with all scans New=0 yet recovery STILL=37) → 0→34 (#26-#29, holds 4 consecutive rebuilds — the AWS-ML +1 from #26 joined the limbo set; byte-identical composition 24/9/1 all four cycles, confirmed 2026-08-05 07:41 with all scans New=0 + watchdog short-circuited yet recovery STILL=38; limbo set = 5 un-ingested: AWS-CB 动态-ip #24, AWS-ML generate-autonomous + automating-customer-retention #20, Interconnects latest-open-artifacts-23 #19, AWS-ML introducing-web-search-foundation-mode #26). Generalized rule: EVERY un-ingested organic article permanently shifts the expected rebuild signature by its delta until promoted to raw/articles/. When all scans show New=0 but recovery remains elevated, that is the persistent-shift steady state — NOT new organic content, NOT a stuck pipeline; verify by reading prior rebuilds' un-ingested-article set. Distinguish persistent-shift from a live cascade by scan New-counts (0 vs >0) and watchdog state (short-circuited vs active). Ingestion-driven −N reverse shift (2026-08-06, rebuild #37 — FIRST observation): when the drain (wiki-inbox-scan-v2 / wechat-inbox-pipeline) ingests limbo articles into raw/articles/ between rebuilds, the next rebuild's recovery per-feed count DROPS by exactly the ingested set (observed: AWS-ML 17→15, baseline 0→38 → 0→36) — recovery.py skips anything already in raw/articles/. A recovery count DECREASE is INGESTION, NOT rotation and NOT feed-health change — the mirror image of the persistence refinement (organic publication shifts the signature UP, ingestion shifts it DOWN). Diagnostic: grep raw/articles/ for the missing filenames (presence confirms ingestion) + cross-reference the drain cron's log line (e.g. "rss 38→0 ... 2 ingest"). Expect the baseline to keep dropping by the ingested set on subsequent rebuilds until the limbo set empties. Drain-consumes-without-ingest carve-out (2026-08-06, rebuild #38): a drain can empty the ENTIRE inbox (36→0) while ingesting ZERO RSS articles — the 06:25 wechat-inbox-pipeline run processed all 36 files as 22 documented premove + 10 step0 + 4 qc skip → 0 ingest, and the baseline held BYTE-IDENTICAL (recovery=40, inbox 0→36, 2nd consecutive rebuild). Drain-log wording like '入档' (observed: 'mobileye/n8n 08-06 入档') means PREMOVE documentation (recognized as known, no LLM), NOT ingestion — never predict a baseline shift from it. Diagnostic discipline: grep raw/articles/ for the limbo filenames BEFORE concluding any shift — presence = ingestion (baseline drops by exactly that set), absence = no shift. Related verification caveat: a grep-based file-set diff against a session reference only extracts filenames that appear as .md tokens in the reference's prose (most inbox files never appear there), so it will show nearly every current file as 'not in reference' — use it ONLY to confirm expected-absent files (the ingested set), and rely on composition counts + per-feed recovery splits + known markers (cascade articles, rotation states) for the real byte-identical check. **Shorthand-name grep trap (hit 2026-08-06 rebuild #39): session-log limbo names are ABBREVIATED (e.g. 'introducing-web-search-foundation-mode' vs on-disk 'introducing-web-search-on-amazon-bedrock-for-foundation-mode') — grepping the shorthand returns 0 matches → false 'absent' signal that could be misread as a baseline shift; grep a distinctive loose token (e.g. 'web-search') or the unambiguous fragment instead. Mirror-image FALSE-POSITIVE variant (hit 2026-08-06 rebuild #44): the loose-token grep can ALSO return a hit that looks like the limbo article but is a different OLD article — 'web-search' matched introducing-web-search-on-amazon-bedrock-agentcore.md in raw/articles/, which at first read looked like the #26 limbo article's ingestion; frontmatter verification (source_url .../agentcore/, ingested 2026-06-19) proved it was an unrelated article sharing the slug prefix. Rule: when a grep hit's filename DIFFERS from the limbo slug, verify frontmatter (source_url + ingested date) BEFORE concluding ingestion — a shared slug prefix is NOT proof; only a matching slug + recent ingested date justifies predicting a baseline shift (absence = no shift, expect the deterministic signature unchanged). Slug-difference TRUE-POSITIVE confirmation (2026-08-07, rebuild #46): the #44 rule was confirmed in the POSITIVE direction — the ingested article's on-disk slug DIFFERED from its inbox/limbo slug (building-an-agentic-app-deployer-with-amazon-bedrock-and-aws-lambda.md vs limbo ...-and-aws.md; recovery slugify ≠ ingest slugify) yet it WAS the ingested #45 cascade article (frontmatter source_url matched + ingested 2026-08-07). Filename mismatch is NOT evidence against ingestion — source_url is the ground truth; dedup_rm=0 at Step 5 (recovery already skipped via URL matching) corroborates. Co-occurring rotation on the same feed as an ingestion shift (2026-08-07, rebuild #46 — FIRST observation): AWS-ML 17→16 looked like pure −1 ingestion, but the file set ALSO rotated: generate-autonomous + automating-customer-retention (#20 limbo) rotated OUT of the feed window (both absent from inbox) while announcing-the-agentic-catalog (19.1KB big) + control-agent-behaviors (10.1KB mid, -new- title-collision dedup suffix) rotated IN; composition 28/9/1 → 26/10/1 — pure ingestion keeps big−N only (27/9/1). Distinguishing signal: ANY composition shape change on an ingestion-shift cycle = co-occurring rotation; corroborate via feed-size shrink (AWS-ML 588KB→563KB) + file-set diff. Corollary: the limbo set can shrink via ROTATION (unrecoverable unless the window rotates back — same fate as AWS-CB 动态-ip #24), not only via ingestion. Stale-DB re-report variant (confirmed 2026-08-05 03:28, rebuild #27): scan New>0 does NOT always mean organic content. If a scan shows New=1 but recovery shows NO delta (per-feed split byte-identical to the prior rebuild — e.g. AWS-ML stayed 13 while scan said New=1), the article is a RE-REPORT of an already-captured un-ingested article whose blogwatcher DB read-state wasn't persisted (same root cause as the watchdog processing gap: URL format mismatch prevents the UPDATE articles SET is_read=1 from matching). Diagnostic evidence to confirm: (a) recovery per-feed split byte-identical to the prior rebuild; (b) watchdog "1 written" is a DUP — curl-recovery's "N URLs already in rss-inbox/" message proves the inbox stayed at recovery's count after watchdog ran (if watchdog had written a net-new file, that count would be recovery+1); (c) watchdog marks 1 read in DB → the re-report typically clears the NEXT cycle (one-cycle echo, not a recurring cascade). Rule: ALWAYS read recovery's per-feed split BEFORE concluding organic — delta=0 + New>0 = stale re-report (no inbox delta, no signature shift); delta=+N + New>0 = genuine cascade (inbox +N per the +N rule). Feed-window rotation carve-out (2026-08-05 12:28, rebuild #30): delta=0 + New>0 can ALSO be ORGANIC — if the recovered FILE SET changed while the per-feed count stayed identical (AWS-CB: quicksight-sso 18.9KB big dropped out of the feed's ~10-article recent window, new weekly roundup aws-一周综述bedrock-中的-gpt-模型降价 6433B mid entered; recovery AWS-CB stayed 7). Distinguishing evidence: watchdog ACTIVE (1 written, NOT short-circuited) + composition shift (24/9/1 → 23/10/1) = organic rotation; watchdog short-circuited + byte-identical file set = stale re-report. NEVER conclude from the count alone — per-feed count identical ≠ byte-identical composition; diff the actual file set against the prior rebuild (roundup feeds like AWS-CB 一周综述 rotate oldest-article-out as new weekly digest publishes, keeping count flat while swapping a big file for a mid one). Rotation RECURRENCE (2026-08-05 16:50, rebuild #32): rotations can run CONSECUTIVELY — #30 rotated (24/9/1→23/10/1), #31 held, #32 rotated AGAIN (new roundup aws-一周综述雅典本地区域... 9391B mid entered while the #30 roundup stayed; the older filename gained a numeric dedup suffix ...等2.md on title collision). Composition counts can hold IDENTICAL across rotations (23/10/1 ×3) — count-identical ≠ set-identical in BOTH directions: equal counts do NOT prove a byte-identical set, and a changed set does NOT imply changed counts. Composition counts can ALSO CHANGE across rotations (2026-08-05 18:55, rebuild #33, 3rd consecutive rotation): 23/10/1 → 24/9/1 (big +1/mid −1) while AWS-CB per-feed count held flat at 7 — a genuinely-NEW big article (140KB Kiro, source_published that day, absent from ALL prior session logs) entered the feed window while the prior roundup rotated OUT. Rotation is NOT limited to roundup-slot swapping — any size-class article can enter/exit the feed's recent ~10-article window, and the feed's total HTTP size shifts accordingly (AWS-CB feed 551KB→580KB as the big article entered). Classification: watchdog ACTIVE + a NEW filename (roundup OR regular article) + flat per-feed count = another rotation cycle (organic, no action), regardless of whether composition counts match or shift vs the prior rebuild; only watchdog short-circuited + byte-identical file set = stale re-report. Never classify on composition-count identity alone — check the actual file set + watchdog state. Re-entry rotation variant (2026-08-05 23:51, rebuild #34, 4th consecutive rotation): watchdog state alone is INSUFFICIENT in BOTH directions — a rotation can occur with watchdog short-circuited (0/5) AND all scans New=0 when the entering article is a RE-ENTRY already marked read in blogwatcher DB (observed: 使用-amazon-athena-分析-kiro-团队用量报表动态模型列 22.9KB, in-window at #26/#27, rotated back in while limbo 动态-ip-场景下的-site-to-site-vpn组网方案 30.6KB rotated out; AWS-CB count held flat at 7). Re-entry rotations do not surface as scan-New (DB read-state already set), so the scan/watchdog path never activates. Full classification matrix: stale re-report (#27) = scan New>0 + recovery delta=0 + byte-identical set; classic rotation (#30/#32/#33) = watchdog ACTIVE + new filename + flat count; re-entry rotation (#34) = all scans New=0 + watchdog short-circuited + CHANGED file set. In ALL cases the file-set diff against the prior rebuild is the ground truth — never classify on watchdog state or counts alone. Final-count reconciliation sanity check: inbox_final = recovery + curl-recovery − case-study_rm − sub-1KB_rm (a watchdog dup contributes 0); a final count off by exactly 1 from this formula usually means the watchdog write was a dup of a recovery file, not a missing file. The Netflix modeling-device-capabilities-for-analytics 3396B small file re-emits every sweep and survives the sub-1KB filter (3.3KB > 1KB) — a stable rebuild feature, not a new artifact (reappeared in 10 rebuilds). | v5.9.23.35+ observations; OUT-absent range 30-35 confirmed v5.9.23.50-54 (transitional); CrewAI-only floor observed v5.9.23.58, confirmed v5.9.23.60-62; fully-drained steady-state recovery=6-10 confirmed v5.9.23.62+; fresh-cycle rebuild confirmed v5.9.23.98, double-confirmed v5.9.23.99 (2026-08-01, two same-day rebuilds with byte-identical signatures); septuple-confirmed 2026-08-02 (7 consecutive rebuilds across 2 days — cross-day determinism established) |
| Case-study filter rm ~30% (100% precision) | inbox >=10 files after recovery | 5-7 files removed per sweep. 100% precision across N=400+ runs. Can use unsupervised. | 62+ runs |
| Trafilatura-upgrade=0 expected | curl-recovery writes 50-60 GDM stubs → sub-1KB filter removes → 0 sub-2KB survivors | 0 is correct outcome, not failure. | Multiple GDM-heavy runs |
| 4-milestone composition (3 stable + 1 rotating, v5.9.23.26 REFINED) | Interconnects + AWS-CN-ML backlog sweep | 3 stable core (farewell-ai2 + OUT mythos + GLM-5.2) + 1 rotating slot (AWS-CB, Interconnects, etc.). Rotation length: 1-2 cycles transient, 3 cycles extended, 5+ semi-stable, 6+ promote to stable. | 11+ consecutive crons |
| blogwatcher-cli 20s timeout = Go HTTP client transient (multi- OR single-feed) | ≥2 feeds TIMEOUT (multi-feed variant) OR one feed TIMEOUT that persists on retry (single-feed variant) | Not feed death. Check recovery+curl-recovery output as ground truth. If recovery writes ≥1 article on TIMEOUT'd feed → definitive feed alive; if recovery writes 0 but curl-recovery shows HTTP=200 + stubs written → content path still intact. Single-feed variant confirmed 2026-07-31 15:53: GDM-only TIMEOUT ×2 with recovery GDM=0/5 skipped, curl-recovery HTTP=200 + 53 stubs (all filtered) — feed fine. Cross-cycle variant (confirmed 2026-08-05, rebuild #26): GDM-only TIMEOUT also REPEATED 2 consecutive cycles (rebuilds #24/#25, 2026-08-04) then cleared on the next cycle (rebuild #26 clean all-7 pass, no timeouts) — treat as day-scale transient, never escalate on 2 consecutive GDM-only timeouts; the clean pass arrives within ~1 cycle of the streak. In-cycle clear variant (2026-08-06, rebuild #40): a GDM-only TIMEOUT can ALSO clear on an IMMEDIATE retry within the same cycle (retry → Found 100, New=0) — always attempt one immediate retry before classifying; same-cycle clear = transient confirmed, zero escalation. | Multiple observations; single-feed variant 2026-07-31 |
| curl-recovery catches all feeds, not just GDM | curl-recovery written >0 from non-GDM feeds | Treat curl-recovery output as ground truth across ALL 7 feeds. It catches articles blogwatcher auto-marked-read before scan. Per-feed HTTP=0 on curl-recovery (FIRST observed 2026-08-06 rebuild #39: AWS-CB feed HTTP=0 size=0 after ~15 consecutive HTTP=200 cycles at 435-580KB) = connection-level transient, NOT feed death — recovery.py's written count for that feed is the ground truth (7 AWS-CB articles captured from feed entries that cycle), curl-recovery's per-feed HTTP status is secondary; no escalation, expect self-heal within ~1-2 cycles (1-cycle self-heal CONFIRMED 2026-08-06 rebuild #40: AWS-CB back to HTTP=200 size=580KB, byte-identical to the pre-transient baseline). | v5.9.20+ confirmed; single-feed HTTP=0 variant 2026-08-06 rebuild #39 |
| Netflix URL health variable (200 or 403) + recovery works = content valid (v5.9.23.59 REFINED; v5.9.23.73 streak update; v5.9.23.78 oscillation confirmed) | Netflix articles checked via curl — all-200 streak (20+ runs) ended at v5.9.23.73 when 4/5 returned 200 and 1 returned non-200. Variable CDN behavior returned. Then all-200 resumed at v5.9.23.78 (5/5 all 200), confirming bidirectional oscillation. | Don't treat EITHER status as "content dead". Signal pattern: feed 200 + articles (403 OR 200) + recovery works = valid content. CDN behavior oscillates between "all 200" and "mostly 200 with 1 non-200" — neither is permanent. The all-200 streak can resume after a variable period. Treat as ongoing oscillation, not a state change. | 6+ crons with 403; 20-run all-200 streak ended v5.9.23.73, all-200 resumed v5.9.23.78 |
| OUT=1 baseline when OUT mythos persists | OUT recovery=0 | OUT=0 does NOT break baseline if OUT mythos persists in inbox from prior cron. If inbox was empty AND OUT=0 → legitimate anomaly (not feed-death). | 40+ crons |
| OUT=0 with empty prior inbox = new baseline when OUT mythos ingested (confirmed v5.9.23.57) | inbox was emptied by wiki-inbox-scan-v2 between this cron and prior run, AND OUT recovery=0 | This is now the new normal, not an anomaly. After the OUT mythos milestone is ingested, OUT recovery=0 persists indefinitely (20+ articles all marked skipped). The "investigate feed health" diagnostic for 2+ consecutive OUT=0 runs only applies when OUT-recovery shows NEW articles being missed — if recovery shows 0 written / 20 skipped, feed health is fine. Recovery drops to ~6-10 (fully-drained steady state) when OUT milestone is absent and all other backlogs are exhausted; the earlier ~30-35 transitional range (v5.9.23.50-54) was driven by residual AWS-CB-ML backlog. | 15 consecutive runs confirmed (v5.9.23.50 → v5.9.23.70) |
| URL-200 health — domain-split reporting | URL health check with per-domain breakdown | Report each domain's ok/fail separately. AWS-only connection-refused is transient cron env artifact (DNS resolution to 127.0.0.1) — retry immediately when AWS-only failures appear; the transient resolves within seconds (observed 7/11 → 11/11 at extreme-drain). CrewAI HTTP 000 (curl connection refused) on individual articles is also transient — not a feed health signal when curl-recovery shows feed-level 200. Mixed-domain failures = real problem — EXCEPT when the failing domains are all documented-transient types AND DNS is healthy (v5.9.23.99 carve-out, broadened 2026-08-01 16:51): the Netflix carve-out (14:38 run: aws 15/16 + netflix 3/4 failing together, dscacheutil healthy, patient retry 28/28) now extends to aws+crewai mixed failures (16:51 run: aws 12/16 + crewai 4/5, dscacheutil healthy CloudFront 3.167.192.33 + Fastly, patient retry 28/28). Netflix fails are EXPECTED CDN oscillation (v5.9.23.59/73/78); CrewAI individual-article 000 is documented transient (v5.9.23.65); AWS connection-level transient is the v5.9.23.92 variant. Generalized diagnostic rule: before escalating ANY mixed-domain failure, run dscacheutil -q host -a name <domain> — healthy real IPs (not 127.0.0.1) + all failing domains ∈ {AWS connection-level, CrewAI 000, Netflix CDN} = patient retry (--connect-timeout 5 --max-time 10) resolves, never escalate. Escalate only on DNS poisoning (127.0.0.1, v5.9.23.88) or failures persisting 2+ consecutive cycles. PERSISTENT variant (v5.9.23.88, 2026-07-31 11:44): AWS-only 0/13 persisted the ENTIRE cycle across immediate retry + sleep 20s + non-sudo dscacheutil -flushcache. Diagnostic: dscacheutil -q host -a name aws.amazon.com returns ip_address: 127.0.0.1 while dig +short aws.amazon.com returns real CloudFront IPs (3.167.192.x) — macOS system resolver cache poisoned (leftover VPN/DNS-override), NOT feed death. Content unaffected: recovery.py captured full 13-38KB from feed entries regardless. Fix (needs user, sudo blocked in cron): sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder. Ad-hoc workaround: forced-IP health check curl --resolve aws.amazon.com:443:<real-ip> (connects fine, returns 404 on /blogs/ paths = reachable). Self-heal confirmed v5.9.23.89 (2026-07-31 13:50): the 0/13 "persistent" cycle was followed ~2h later by 13/13 all reachable WITHOUT sudo intervention — treat the persistent variant as day-scale transient; do NOT escalate to user for a single affected cycle, escalate only if AWS-only 0/N repeats across 2+ consecutive cycles. Connection-level variant (v5.9.23.92, 2026-07-31 21:30): first-pass URL-200=20/21 (aws 13/14); immediate retry showed failures SPREADING 1→5×000 while dscacheutil -q host -a name aws.amazon.com returned HEALTHY CloudFront IPs (3.167.192.x, NOT 127.0.0.1) — distinct from the poisoning variant. Patient check (curl --connect-timeout 5 --max-time 10) showed all 14 AWS URLs reachable; self-resolved within minutes. Diagnostic rule: ALWAYS run dscacheutil BEFORE concluding poisoning — healthy DNS + 000s = connection-level transient, retry with a patient connect timeout, never escalate. Network-egress variant (2026-08-03 18:46, rebuild #18): aws first-pass 3/16 → 0/16 on patient retry with LibreSSL SSL_connect: SSL_ERROR_SYSCALL at connect, DNS healthy (CloudFront 3.167.192.10), AND www.google.com also timing out → whole-network egress transient, NOT AWS-specific feed death. Tiebreaker when retries keep failing: forced-IP check curl --resolve aws.amazon.com:443:<real-ip> returning HTTP=200 proves HTTP-layer reachability — combined with recovery.py having captured full article content from feed entries, declare content valid and skip escalation. Distinguish the AWS-only variants by signature: poisoning (v5.9.23.88) = dscacheutil returns 127.0.0.1; connection-level (v5.9.23.92) = healthy DNS + spreading 000s that self-resolve in minutes; network-egress (#18) = healthy DNS + SSL_ERROR_SYSCALL + unrelated domains (google.com) also unreachable — this one may NOT self-resolve within the cycle; forced-IP 200 is the proof of life. Resolver-path variant (2026-08-04 15:17, rebuild #24): healthy DNS (dig real CloudFront 18.65.14.x; dscacheutil CNAME dr49lng3n1n2s.cloudfront.net, NOT 127.0.0.1) + suspicious fake IPv6 1100:2200:3300:4400:5500:6600:7700:8800 in dscacheutil (sequential placeholder pattern — the tell that the SYSTEM RESOLVER path is the broken link, not AWS) + normal-resolution curl → 000 that PERSISTS through cycle, patient retry, AND IPv4-forced curl -4 — BUT forced-IP curl --resolve aws.amazon.com:443:<real-ip> → HTTP=200 in ~0.6s on BOTH the blog index and the actual article URL, while all non-AWS domains are 200 (NOT network-egress). Key technique: --resolve bypasses the system resolver entirely, so a forced-IP 200 (or even 404 on a bogus path) proves HTTP-layer reachability even when -4 fails — a failed -4 does NOT prove unreachability when a fake-IPv6 record is poisoning the resolver path. Single-cycle AWS-only failure with proven reachability + feed-captured content = no escalation; revisit if 0/N repeats 2+ consecutive cycles. Self-heal confirmed 2026-08-04 17:25 (rebuild #25): the very next cycle was 33/33 FIRST pass with aws 19/19 ~2h later — the resolver-path variant self-healed within hours, same day-scale trajectory as the other three variants; ALL FOUR AWS transient variants now have confirmed ≤~5h self-heal, never escalate on a single affected cycle. Self-heal confirmed 2026-08-03 23:21 (rebuild #19): the very next cycle was 30/30 FIRST pass with zero transient dips — the network-egress variant resolved within ~5h, same day-scale trajectory as the poisoning variant; never escalate for a single affected cycle. | 41+ runs; CrewAI 000 confirmed transient v5.9.23.65; persistent AWS-DNS variant v5.9.23.88; self-heal confirmed v5.9.23.89; network-egress variant 2026-08-03 rebuild #18 |