ai-digest

name	ai-digest
description	Run the daily AI tech news digest pipeline — collect, deduplicate, summarize, deliver

AI Digest — Daily Tech News Pipeline

You are running the ai-digest pipeline. Follow these steps exactly.

Progress Logging

After completing each step, log a progress line to logs/YYYY-MM-DD.md via Bash:

echo "[$(date '+%Y-%m-%d %H:%M')] [pipeline     ] <message>" >> logs/YYYY-MM-DD.md

This keeps the user informed in real-time (they watch the log via tail -f). Log at every step transition — do not batch multiple steps without logging.

Step 0: Validate Sources

Before anything else, invoke the /validate-sources skill. It checks config/sources.yml for structural errors and broken URLs.

• If there are structural errors (missing keys, wrong types, invalid format) — stop and report. The pipeline cannot run with a broken config.
• If there are broken URLs (404, connection errors) — log the warnings but continue. The pipeline will simply get no data from those sources.

Log the validation summary to logs/YYYY-MM-DD.md using the format:

[YYYY-MM-DD HH:MM] [validation   ] Sources validation: X passed, Y failed, Z unable to verify
[YYYY-MM-DD HH:MM] [validation   ] ✗ rss: source-name — HTTP 404

Log each failed/unverified source as a separate line.

Step 1: Read Configuration

Read these files:

• config/sources.yml — list of all sources
• config/delivery.yml — language, output path, notification settings
• CLAUDE.md — personal context (stack, projects, interests, topics to ignore)

Log: Step 1: Config loaded (X RSS sources, Y GitHub repos)

Step 2: Read Previous Digests (for deduplication)

Call the fetch_previous_urls MCP tool (no parameters). It reads the last 3 days of digest markdown files from the output path and returns all previously published URLs and titles.

Save the returned entries — you will use them for semantic deduplication in Step 4.

Log: Step 2: Previous digests loaded — X entries from Y days or Step 2: No previous digests found, skipping dedup

Step 3: Collect Data

Launch sub-agents in parallel to collect data from all sources:

Agent 1 — RSS feeds: Call the fetch_all_rss MCP tool (no parameters). It fetches all RSS feeds in parallel internally. Collect results.

Agent 2 — GitHub Releases: Call the fetch_github_releases MCP tool (no parameters). Collect results.

Merge all DigestItem[] arrays from all agents into one list.

Log: Step 3: Collected X RSS items (Y sources) + Z GitHub releases

Step 4: Deduplicate

4a: Programmatic cross-day deduplication

Call the check_duplicates MCP tool with all collected items from Step 3. Pass items as: { "items": [{ "title": "...", "url": "...", "source": "..." }, ...] }

The tool returns each item classified as exact_duplicate, likely_duplicate, or unique.

• Remove all items marked exact_duplicate — no exceptions.
• Items marked unique — keep as-is.

4b: Review likely duplicates

For each likely_duplicate item, compare with the matched_with entry:

• If the item contains substantially new information (new version, breaking change, new analysis, different perspective) — KEEP it.
• If it covers the same topic without new information — REMOVE it.
• When in doubt — REMOVE. Fresh content over repeats.

4c: Semantic deduplication (your judgment)

Review remaining unique items against the entries returned in Step 2. If you notice a topic that was already covered in a previous digest and the new item adds nothing substantial — remove it.

4d: Within-day deduplication

Among the remaining items:

• Merge items with identical URLs from different sources into one entry
• Merge items with very similar titles about the same topic into one entry
• If multiple releases of the same package/tool appear, collapse into one entry with the latest version

Log: Step 4: Deduplicated — removed X exact, Y likely, Z semantic, merged W within-day. V items remaining

Step 5: Filter

• Read "topics to ignore" from CLAUDE.md
• Remove any items matching ignored topics (e.g. crypto, NFT, blockchain, web3)

Log: Step 5: Filtered — removed X items (topics: Y), Z items remaining

Step 6: Categorize and Summarize

Read the language setting from delivery.yml. Generate ALL text in that language.

Assign each item to one category:

• Hot — the 3-5 most important, impactful, or trending news of the day across all topics. Major announcements, breaking changes, viral discussions. This category is mandatory — it must always be present, even if other categories are empty. Pick items that a tech professional absolutely cannot miss today.
• Relevant to Your Projects — items that match your stack, active projects, or interests from CLAUDE.md. Add a note explaining the connection.
• AI / LLM — AI models, tools, agents, LLM news
• Frontend — React, CSS, browser APIs, UI frameworks
• DevTools / Releases — developer tools, package releases, CLI tools

An item placed in Hot should NOT be duplicated in other categories.

For each item, write a catchy 1-2 sentence summary as the headline. Make it engaging, not dry. The headline should make the reader want to click through.

Log: Step 6: Categorized — Hot: X, Relevant: Y, AI: Z, Frontend: W, DevTools: V

Step 7: Generate Markdown

Create the digest file with this structure:

---
date: YYYY-MM-DD
type: digest
language: <language from config>
sources: <number of sources that returned data>
items: <total items after dedup>
---

# <Digest title in configured language> — <date in configured language format>

> <total items> items from <sources count> sources

<intro paragraph>

## 🔥 Hot
- **<catchy headline>** — <summary>. [<source>](url)

## 🎯 <"Relevant to Your Projects" in configured language>
- **<catchy headline>** — <summary>. [<source>](url)
  _<relevance note>_

## 🤖 AI / LLM
- **<catchy headline>** — <summary>. [<source>](url)

## ⚛️ Frontend
- ...

## 🔧 DevTools / Releases
- ...

Intro paragraph

Before the categorized sections, write a narrative intro (2–4 short paragraphs) that:

• Synthesizes the Hot items into a cohesive narrative — identify themes, connections, or why today's news matters
• Uses an editorial, opinionated tone — not a dry summary but a take: what's exciting, what's worth watching, what might break things
• Explains why these items are significant, not just what happened
• Ends with a natural transition into the full digest
• Written in the configured language
• Plain prose, no headers, no bullet points, no emoji
• Break into short paragraphs (1–2 sentences each) for easy scanning — avoid walls of text

The Hot category is mandatory and must always be present. Skip any other category that has zero items.

Step 8: Write File

Write the generated markdown to: <output_path>/YYYY-MM-DD.md Use today's date for the filename.

Log: Step 8: Written to <output_path>/YYYY-MM-DD.md

Step 9: Notify

If notification is true in delivery.yml, send a macOS notification:

osascript -e 'display notification "<N> items from <M> sources" with title "AI Digest Ready"'

If the pipeline produced zero items (all sources failed), send:

osascript -e 'display notification "All sources failed. Check logs." with title "AI Digest Failed"'

Step 10: Log Summary

Print a summary of this run:

• Sources attempted and items collected per source
• Any warnings from tools
• Total items after deduplication
• Output file path