newsletter-second-brain

name	newsletter-second-brain
description	Ingest AI newsletters from ClawEmail into a local wiki, extract reusable workflows, and produce bilingual daily digests. Use when operating a newsletter-driven second-brain setup with mail-cli, Hermes, and markdown notes.
version	1.0.0
author	Hermes Agent
license	MIT
metadata	{"hermes":{"tags":["newsletter","clawemail","wiki","digest","bilingual","workflow","second-brain"]}}

Newsletter Second Brain

Use this skill when the user has a ClawEmail/newsletter mailbox and wants to:

• ingest newsletter emails into a markdown wiki
• extract reusable workflow patterns from newsletter content
• generate bilingual daily digests
• turn noisy email HTML into durable knowledge artifacts

When this skill is useful

• Beehiiv/Substack newsletters are arriving in a ClawEmail sub-mailbox
• raw mail-cli read body output is noisy and too HTML-heavy
• the user wants a second-brain workflow, not just ad hoc summaries
• the user wants both source summaries and higher-level concept/workflow extraction
• the user wants Chinese + English digest output

Key finding from real usage

Do NOT rely on naive HTML stripping from mail-cli read body for newsletter ingest. It often preserves footer junk and can miss the true article structure. A dedicated parser that walks headings/paragraphs/list items produces much better source notes.

Required ingredients

• mail-cli configured with a newsletter profile
• a local wiki directory (for example ~/wiki)
• WIKI_PATH and ideally OBSIDIAN_VAULT_PATH pointing at that wiki
• a parser script such as ~/.hermes/scripts/newsletter_extract.py

Project-vault routing (important)

If the user already has a multi-vault project wiki, prefer the project's canonical vault structure over a standalone ~/wiki bootstrap directory.

In real usage on this machine, the better long-term destination was:

• canonical vault: /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global
• pages under: /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/wiki/
• raw sources under: /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/raw/

For cross-project AI/newsletter/workflow knowledge, route into _global rather than a project-specific vault. If you bootstrap in ~/wiki first, either migrate into the canonical vault later or make ~/wiki a symlink to that canonical location.

Recommended directory shape

~/wiki/
  SCHEMA.md
  index.md
  log.md
  raw/articles/
  concepts/
  queries/
  workflows/

Proven workflow

1. Inspect the inbox

Use mail-cli --profile newsletter mail list --fid INBOX --limit 20 --json to identify candidate emails.

Prioritize items about:

• frontier model launches or rankings
• agent workflows
• operator playbooks
• evaluation shifts
• deployment or infrastructure changes

2. Extract the body with a parser, not just a text dump

Preferred pattern:

python ~/.hermes/scripts/newsletter_extract.py <message-id>

The parser should:

• call mail-cli --profile newsletter read body --id <id> --json
• parse the HTML body with BeautifulSoup
• walk h1/h2/h3/h4/p/li
• normalize whitespace
• remove footer noise like unsubscribe/promotional boilerplate
• preserve section structure as {heading, items}

Important: this worked much better on Beehiiv/Substack mail than raw text extraction.

3. Write a structured raw source page

Save a parsed snapshot to raw/articles/YYYY-MM-DD-source-slug.md with frontmatter like:

---
source_url: unknown
ingested: YYYY-MM-DD
updated: YYYY-MM-DD
source_type: newsletter-email-parsed
sender: [e-mail]
subject: Subject line
email_date: YYYY-MM-DD HH:MM:SS
message_id: <mail id>
extractor: ~/.hermes/scripts/newsletter_extract.py
---

Then store sectioned markdown under ## Parsed structure.

4. Create a source summary page

Save a summary to queries/<slug>-summary.md.

The best summary format in practice was:

• Source
• Main point
• Key takeaways
• Reusable workflow signals
• Why this matters for the second-brain
• Follow-up actions

If extraction quality is partial, say so explicitly instead of pretending to have full coverage.

5. Extract patterns, not just headlines

A key lesson from this workflow: newsletter value often compounds more from reusable workflow examples than from the top headline.

Promote repeated patterns into dedicated concept/workflow pages such as:

• planner/executor model handoff
• morning newspaper / ranked briefing workflow
• document processing with verification

6. Produce a bilingual daily digest

Use a strict bilingual template. Keep both English and Chinese for every major section.

Recommended structure:

## English
### Selection note
### 1. [signal]
- what happened
- why it matters
- watch next
### 2. [signal]
...
### Reusable workflow ideas

## 中文
### 选择说明
### 1. [信号]
- 发生了什么
- 为什么重要
- 后续观察点
### 2. [信号]
...
### 可复用 workflow 想法

## Bottom line / 总结

7. Keep navigation updated

Every ingest should update:

• index.md
• log.md

If you create concept/workflow pages, add them to the appropriate index section as well.

Useful file conventions

• source raw page: raw/articles/YYYY-MM-DD-source-topic.md
• digest page: queries/daily-newsletter-digest-YYYY-MM-DD.md
• standalone query/summary page: only create queries/source-topic-summary.md for genuinely heavyweight/high-signal items worth separate display; otherwise fold newsletter items into the daily digest
• workflow page: workflows/<workflow-name>.md
• concept page: concepts/<concept-name>.md

Query-directory hygiene

For this user's _global newsletter workflow, keep wiki/queries/ root sparse. The user now prefers Chinese and English digest content archived separately, not as one long combined bilingual page.

Default daily layout:

• English archive: wiki/queries/daily-newsletter-digest/en/daily-newsletter-digest-YYYY-MM-DD.en.md
• Chinese archive: wiki/queries/daily-newsletter-digest/zh/daily-newsletter-digest-YYYY-MM-DD.zh.md
• Avoid creating a root combined daily-newsletter-digest-YYYY-MM-DD.md unless the user explicitly asks for a single combined view.

For daily archives, keep generated markdown free of read_file display prefixes such as 12|..., verify no root combined file was created, and verify newsletter raw snapshot frontmatter sha256 against the exact body after the closing frontmatter delimiter.

The daily digest pair should cover the same underlying sources:

• ClawEmail newsletter sources for AI/agent learning
• AI HOT as the primary structured Chinese AI radar source via ~/.hermes/scripts/aihot_context.py; preserve sourceName/source and sourceUrl/url attribution for every AI HOT item, and treat AI HOT summaries as LLM-generated aggregation that should be verified against original URLs before citation
• 量子位 and 机器之心 only as optional supplementary / cross-check sources when available
• 机器之心 via ~/.hermes/scripts/fetch_cn_ai_sources.py, which uses Jina Reader (https://r.jina.ai/http://r.jina.ai/http://https://www.jiqizhixin.com/articles) because direct /, /articles, and /rss often return a data-service shell in browser/cron contexts
• explicit failure/status note only if an expected helper/source cannot retrieve items

Feishu-facing daily newsletter documents should be Chinese-only, with English archives remaining in the wiki. Separate standalone pages are reserved for heavyweight items that deserve independent tracking.

Chinese-only Feishu reading document pattern

When the daily digest cron asks for a Feishu docx reading version, create one Chinese-only document titled 每日 AI Newsletter Digest｜YYYY-MM-DD via Feishu docx OpenAPI:

1. Load FEISHU_APP_ID and FEISHU_APP_SECRET from /Users/leiyuxuan/.hermes/.env without printing secrets.
2. Get a tenant token with POST https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal.
3. Create the doc with POST https://open.feishu.cn/open-apis/docx/v1/documents and payload {"title": "每日 AI Newsletter Digest｜YYYY-MM-DD"}.
4. Write text blocks with POST /open-apis/docx/v1/documents/:document_id/blocks/:document_id/children?document_revision_id=-1.
5. Use real Feishu docx heading/list/quote blocks so the left-side Feishu outline is generated; do not merely type numbered headings in paragraph blocks. For Daily AI Newsletter Digest, align with the “AI 内容结构化阅读版” style:
   • H1 1. 今日总览; H2 1.1 一句话结论, 1.2 三条主线, 1.3 今天建议阅读顺序
   • H1 2. Newsletter 主线：从模型能力到工作系统; H2/H3 for source, technical points, Hermes judgment
   • H1 3. AI HOT 高信号雷达; group items by theme instead of a flat news feed, and explicitly down-rank low-signal items
   • H1 4. 我的判断与后续跟进
   • H1 5. 来源 attribution 与归档 Feishu API block mapping: H1 block_type:3/heading1, H2 4/heading2, H3 5/heading3, paragraph 2/text, bullet 12/bullet, ordered 13/ordered, quote 15/quote.
6. Use the returned document_id to form a stable URL. If FEISHU_DOMAIN in .env is just feishu, normalize the final report URL to https://feishu.cn/docx/<document_id> rather than feishu/docx/<id>.
7. Verify the document by listing its children through the docx API if the Feishu doc read tool is unavailable. Do not fail the digest if sharing/collaborator APIs are unavailable; mention the permission caveat only if access may require manual sharing.

Recommended cleanup layout:

• wiki/queries/ root: sparse navigation/index or exceptional query pages only.
• wiki/queries/daily-newsletter-digest/en/: daily English digest archive.
• wiki/queries/daily-newsletter-digest/zh/: daily Chinese digest archive.
• wiki/queries/heavyweight/: exceptional standalone query pages for high-signal topics worth long-term separate display.
• wiki/queries/archive/: ordinary historical summaries, early experiments, self-tests, and pages superseded by daily digests.
• wiki/status/: operational status files such as newsletter-confirmation-monitor-latest.md; do not keep status/monitor files in queries/.

When reorganizing, also update index.md, log.md, wiki/workflows/newsletter-operations-map.md, relevant wikilinks, and any cron prompts that write to moved paths.

Historical bilingual-digest split migration pattern

When the user wants old combined bilingual daily digests separated by language:

1. Inventory root combined files under wiki/queries/daily-newsletter-digest-*.md.
2. For each file, parse sections by headings:
   • ## English through before ## 中文 → English archive.
   • ## 中文 through before ## 中文 AI 资讯简报 or ## Bottom line / 总结 → Chinese archive.
   • ## 中文 AI 资讯简报（量子位 / 机器之心） stays in the Chinese archive.
   • ## Bottom line / 总结 is split into English bottom line and Chinese summary when possible.
3. Write archives to:
   • wiki/queries/daily-newsletter-digest/en/daily-newsletter-digest-YYYY-MM-DD.en.md
   • wiki/queries/daily-newsletter-digest/zh/daily-newsletter-digest-YYYY-MM-DD.zh.md
4. Move the old combined file to wiki/queries/archive/combined-daily-newsletter-digest/ instead of deleting it, so old links can be recovered.
5. Update index.md to list .en and .zh archive entries, not root combined links.
6. Append a log.md entry documenting the dates processed, archive roots, and legacy combined-file location.
7. Verify:
   • wiki/queries/ root has no daily-newsletter-digest-YYYY-MM-DD.md files.
   • English count equals Chinese count.
   • Each split date has a legacy combined file in archive/combined-daily-newsletter-digest/.
   • No read-display line prefixes such as 1| were introduced into index/log/generated pages.

Digest automation pattern

A good cron setup used two pieces:

1. A pre-run context script like newsletter_digest_context.py that outputs the recent inbox list as compact JSON.
2. A cron prompt that:
   • selects top 1-3 items
   • uses newsletter_extract.py for each selected message
   • writes a bilingual digest
   • updates log.md

If you want consistent quality, force the cron output to follow a saved digest template file.

Critical cron finding

Hermes cron jobs do not fire automatically unless the Hermes gateway is actually running.

On macOS, the reliable pattern was:

• install/load the gateway service
• verify hermes cron status says the gateway is running
• only then trust scheduled digest execution

A useful self-test is to create a one-shot cron job that writes a tiny file into the target wiki directory. This isolates scheduler health from digest complexity.

Use explicit canonical paths in cron prompts

If the wiki has a canonical project-vault path, do not rely on ~/wiki in the cron prompt just because a symlink exists. Use explicit output paths such as:

• digest: /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/wiki/queries/daily-newsletter-digest-YYYY-MM-DD.md
• template: /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/wiki/workflows/bilingual-daily-digest-template.md
• log: /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/log.md
• raw sources: /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/raw/articles/

This avoids hidden dependence on compatibility symlinks and makes the automation easier to reason about.

Confirmation-monitor automation

If you also run a confirmation-monitor cron job, do not leave it on deliver=origin without a resolved target. A more reliable pattern is to make it write to a fixed status file in the canonical vault, for example:

• /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/wiki/status/newsletter-confirmation-monitor-latest.md

The monitor should overwrite that file on each run with:

• scan time
• matching confirmation / verification emails
• which items still need action
• any links that should be opened manually
• a short Chinese summary

This makes the monitor inspectable even when no chat-origin delivery target exists.

User-facing delivery and latest-content checks

For the daily digest, deliver=local is good for silent wiki-only operation, but if the user expects the digest to appear in the current chat, update the cron job to deliver=origin or send a short confirmation with send_message after verifying the file. For this machine, daily-newsletter-digest is configured with deliver=feishu; successful delivery can also be verified by a send_message/cron message_id and last_delivery_error: null.

When the user asks where to see today's latest content:

1. Check current date/time with date. Do not assume the date from prior turns; in long sessions the calendar day may have changed.
2. Search the canonical vault for *YYYY-MM-DD*.md, especially wiki/queries/daily-newsletter-digest-YYYY-MM-DD.md, wiki/everlyme/daily-lens/Everlyme-Daily-Lens-YYYY-MM-DD.md, and raw/articles/YYYY-MM-DD-*.md.
3. Check cronjob list for the digest job's last_run_at, last_status, next_run_at, and last_delivery_error. Also check related daily jobs such as everlyme-daily-lens when the user asks “今日该完成的任务”.
4. If the scheduled digest appears to have been fast-forwarded or missed and no current-day digest exists, run the digest job manually with cronjob run <job_id>.
5. Wait/poll for the output file rather than trusting cron status alone. A manual run can take several minutes; poll the canonical output file and cron session file under ~/.hermes/sessions/session_cron_<job_id>_YYYYMMDD_*.json until a final assistant response or the file appears.
6. Read the generated digest before replying, then push a short user-facing summary to Feishu if the user asked for push/delivery.

Important nuance: cronjob run may schedule an immediate/near-future run, and last_run_at / last_status may lag or remain unset while the output file has already been created. Treat the canonical vault file and log entry as the source of truth for content availability, not only the cron status fields.

Manual recovery when daily cron tasks fail

If daily-newsletter-digest or a companion daily task fails with last_activity=initializing / inactivity timeout, recover the day's artifacts manually instead of only reporting failure:

1. Use date to determine the exact target day.
2. Run python ~/.hermes/scripts/newsletter_digest_context.py to inspect newsletter candidates.
3. Run python ~/.hermes/scripts/fetch_cn_ai_sources.py YYYY-MM-DD and verify the raw Chinese snapshot has a valid sha256 in frontmatter; if absent, add a body hash over content after the closing frontmatter delimiter.
4. For selected newsletter messages, run ~/.hermes/hermes-agent/venv/bin/python ~/.hermes/scripts/newsletter_extract.py <message-id> so the same environment used by cron is exercised. Save raw snapshots under _global/raw/articles/YYYY-MM-DD-<slug>.md.
5. Write split daily digest archives, not a root combined page: _global/wiki/queries/daily-newsletter-digest/en/daily-newsletter-digest-YYYY-MM-DD.en.md and _global/wiki/queries/daily-newsletter-digest/zh/daily-newsletter-digest-YYYY-MM-DD.zh.md; update _global/index.md and _global/log.md.
6. If everlyme-daily-lens also failed, create _global/wiki/everlyme/daily-lens/Everlyme-Daily-Lens-YYYY-MM-DD.md, update wiki/everlyme/Everlyme Daily Lens Index.md, and append a log entry. Be careful not to rewrite files using line-numbered read_file output; use patch or clean Path.read_text()/write_text().
7. Verify files with search_files + read_file, then send a concise Feishu summary with the file paths and main themes.

Pitfall: API keys or strings beginning with sk- may appear in chat during debugging. Treat them as secrets: do not repeat, store, or use them; tell the user to revoke/rotate the key. Important nuance: cronjob run may schedule an immediate/near-future run, and last_run_at / last_status may lag or remain unset while the output file has already been created. Treat the canonical vault file and log entry as the source of truth for content availability, not only the cron status fields. If a manual run is in progress and cronjob list has not updated yet, inspect ~/.hermes/sessions/session_cron_<job_id>_YYYYMMDD_*.json to see whether the cron agent is actively making tool calls, writing files, or stuck; poll both the expected output file and that session file before declaring failure.

A real failure mode: the daily digest cron can show last_status: error and no output file because the cron agent idled during initialization before any tool call, e.g. Job 'daily-newsletter-digest' idle for 786s (inactivity limit 600s) | last_activity=initializing | iteration=0/90 | tool=none. In that case, content was not generated at all, so Feishu delivery cannot have happened. Distinguish this from delivery-only failures such as Feishu/Lark DNS/WebSocket errors, where content may exist locally but sending failed.

Operations-map page and user-facing vault shortcut

For day-to-day use, maintain a single workflow entry page inside the canonical vault, for example:

• /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/wiki/workflows/newsletter-operations-map.md

That page should list:

• canonical vault location
• convenient user-facing shortcut vault location
• digest entry points
• confirmation-monitor output file
• reusable concept/workflow pages
• automation scripts
• cron job IDs and schedules
• first-use checklist

For this machine, the user-facing Obsidian/Finder shortcut is:

• /Users/leiyuxuan/Documents/PyCharm_Project/wiki/ai-news-vault → symlink to /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/wiki
• Home page: /Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global/wiki/Home.md

Use this shortcut for daily browsing, while keeping cron prompts and automation writes pointed at the canonical full paths under _global. Raw sources, index.md, and log.md remain in the full vault root (/Users/leiyuxuan/Documents/PyCharm_Project/wiki/_global), not inside the shortcut.

A single operations-map page plus a named shortcut vault drastically reduces the cognitive overhead of resuming the system after a break.

Pitfalls

• Raw email text dumps are often noisy and misleading.
• Beehiiv newsletter bodies often contain a lot of footer clutter; trim aggressively.
• TLDR AI emails can defeat or under-fill newsletter_extract.py: the script may return only sparse sections while mail-cli --profile newsletter read body --id <id> --json still contains the real newsletter body. If the structured extractor output is too short or missing expected headings, fall back to mail-cli read body --json, strip <style>/<script>/<head> blocks, insert line breaks around block tags (p/div/tr/td/h*/li/a/br), then save a raw snapshot with an explicit “partial/noisy extraction” note.
• When falling back from the parser, do not shell-pipe raw email content into Python. Use execute_code with subprocess.run([...], capture_output=True, text=True) and parse/clean in-process.
• Some newsletters preserve the headline but bury the important workflow section lower down — don’t stop at the first story.
• Token price alone is not enough when summarizing model releases; task-level cost and infrastructure implications matter too.
• Separate source summary pages from pattern pages. Don’t cram both into one file.

What made this workflow work well

• Switching from naive email-text ingest to structured HTML section extraction
• Saving parsed raw pages before writing summaries
• Treating workflow examples as first-class knowledge
• Writing bilingual digest output so the notes are directly usable by the user