mo-feed

name

mo-feed

description

Tech intelligence pipeline for developers. Fetches from Twitter, Hacker News, GitHub, arXiv, HuggingFace, Lobsters, and Reddit in parallel. Classifies content by customizable P0-P3 priority tiers. Renders TUI cards in terminal. Ingests into markdown knowledge base (~/.secondmem/knowledge/). Tracks read/unread status. Use when user says "catch me up", "what's interesting", "daily digest", "feed me", "mo feed", "tech scan", "what's new", "fetch and ingest", or invokes /mo-feed.

mo-feed — Tech Intelligence Pipeline for Developers

One skill. Fetch from everywhere. Classify. Display cards. Ingest to memory. Done.

Twitter ──┐
HN ───────┤
GitHub ───┤──→ Classify (P0-P3) ──→ TUI Cards ──→ Knowledge Base Ingest
arXiv ────┤
HF ───────┤
Lobsters ─┘

What It Does

1. FETCH — pulls from 7+ sources in parallel (Twitter timeline, HN, GitHub trending, arXiv, HuggingFace, Lobsters, Reddit)
2. CLASSIFY — sorts every item into P0-P3 priority tiers using keyword matching
3. DISPLAY — renders beautiful TUI cards in terminal, grouped by tier, sorted by engagement
4. TRACK — maintains a reading list with unread/read status (~/.secondmem/reading-list.md)
5. INGEST — writes P0 + high-signal P1 items into a markdown knowledge base (~/.secondmem/knowledge/)
6. INDEX — updates a chronological timeline for date-based recall (~/.secondmem/timeline.md)

When to Use

• "catch me up" / "what's interesting" / "daily digest" / "feed me"
• "what's new in [topic]" / "scan [source]" / "tech scan"
• "fetch and store" / "fetch and ingest" / "mo feed"
• Any request that implies: get content from internet + show it + remember it

Prerequisites

# Required: headless-twitter for Twitter fetching (Chrome CDP, no API keys)
which headless-twitter || npm install -g headless-twitter

# Required: jq for JSON processing
which jq || sudo apt install jq

# Knowledge base directory (auto-created on first run if missing)
mkdir -p ~/.secondmem/knowledge ~/.secondmem/logs

Pipeline — Execute in Order

Phase 1: FETCH

Fetch from multiple sources in parallel. Default: Twitter timeline + top cross-reference sources. If user specifies a topic, add targeted searches.

Default fetch (no topic specified):

# Twitter timeline (primary) — requires headless-twitter + logged-in Chrome
headless-twitter twitter timeline '' 50 --lang en --json

# Cross-reference sources (parallel curl — no auth needed)
curl -s "https://hn.algolia.com/api/v1/search?tags=show_hn&numericFilters=points>30" | jq '.hits[0:10]'
curl -s "https://huggingface.co/api/models?sort=trending&limit=10" | jq '.[] | {modelId, likes, pipeline_tag, lastModified}'
curl -s "https://lobste.rs/hottest.json" | jq '.[0:10]'
curl -s "https://api.github.com/search/repositories?q=stars:>50+created:>$(date -d '3 days ago' +%Y-%m-%d)&sort=stars&per_page=10" | jq '.items[] | {full_name, stargazers_count, language, description}'

Topic-specific fetch (user says "what's new in [topic]"):

Topic	Twitter Search	Extra Sources
Agentic AI	"agent framework" OR "agent SDK" OR "MCP" OR "tool-use"	arXiv cs.AI, HuggingFace, r/LocalLLaMA
ML/AI	"open source model" OR "fine-tuning" OR "inference" OR "transformer"	arXiv cs.LG, HuggingFace, r/machinelearning
GPU/Hardware	"GPU" OR "CUDA" OR "RISC-V" OR "FPGA" OR "NVIDIA"	GitHub C/C++/CUDA trending, Lobsters hardware
Databases	"database" OR "B-tree" OR "storage engine" OR "PostgreSQL" OR "SQLite"	Lobsters databases, HN, r/programming
System Design	"distributed" OR "latency" OR "consensus" OR "architecture"	Lobsters, HN top, r/experienceddevs
Infra	"microVM" OR "eBPF" OR "kubernetes" OR "Docker" OR "CLI tool"	GitHub Go/Rust trending, Show HN
Startups	"just shipped" OR "just launched" OR "YC" OR "users"	Product Hunt, Show HN
Frontend	"React" OR "Next.js" OR "Svelte" OR "CSS" OR "web components"	GitHub trending JS/TS, r/webdev
Security	"CVE" OR "zero-day" OR "vulnerability" OR "exploit" OR "pentest"	Lobsters security, r/netsec
DevOps	"terraform" OR "ansible" OR "CI/CD" OR "deployment" OR "SRE"	HN, Lobsters devops

Source-specific fetch (user says "scan [source]"):

Source	Command
Twitter	headless-twitter twitter timeline '' 50 --lang en --json
Twitter search	headless-twitter twitter search "QUERY" 30 --lang en --json
Twitter user	headless-twitter twitter user "@handle" 20 --lang en --json
HN top	curl -s "https://hn.algolia.com/api/v1/search?tags=front_page" | jq '.hits[0:15]'
HN Show	curl -s "https://hn.algolia.com/api/v1/search?tags=show_hn&numericFilters=points>30" | jq '.hits[0:15]'
GitHub trending	curl -s "https://api.github.com/search/repositories?q=stars:>50+created:>$(date -d '3 days ago' +%Y-%m-%d)&sort=stars&per_page=15" | jq '.items[]'
arXiv AI	curl -s "http://export.arxiv.org/api/query?search_query=cat:cs.AI+OR+cat:cs.LG&sortBy=submittedDate&sortOrder=descending&max_results=10"
HuggingFace	curl -s "https://huggingface.co/api/models?sort=trending&limit=10" | jq '.[]'
Lobsters	curl -s "https://lobste.rs/hottest.json" | jq '.[0:15]'
Reddit	curl -s "https://www.reddit.com/r/programming/hot.json?limit=15" -H "User-Agent: mo-feed/1.0" | jq '.data.children[].data'
r/LocalLLaMA	curl -s "https://www.reddit.com/r/LocalLLaMA/hot.json?limit=15" -H "User-Agent: mo-feed/1.0" | jq '.data.children[].data'

Phase 2: CLASSIFY

Classify ALL fetched content into priority tiers using keyword matching.

Default Priority Tiers

Customization: Edit the tiers below to match YOUR interests. These defaults are tuned for a full-stack developer interested in AI, systems, and infrastructure. Swap keywords, rename tiers, add new domains — the classifier just does keyword matching against text + author.

Tier	Domain	Keywords / Signals
P0	Agentic AI	agent, MCP, tool-use, memory system, harness, skills, RAG, Claude, LangChain, CrewAI, AutoGen, agentic, deepagent
P0	Foundational AI/ML	transformer, attention, training, distill, inference, RLHF, DPO, open-source model, vLLM, TRL, GGUF, quantiz, fine-tun, benchmark, LLM, GPT, param, token
P1	GPU / Hardware	GPU, CUDA, NPU, RISC-V, FPGA, chip, silicon, NVIDIA, AMD, robot, hardware, spacecraft
P1	System Design	database, B-tree, LSM, consensus, distributed, latency, throughput, architecture, CAP, CRDT, system design, query optim
P1	Infra Engineering	microVM, sandbox, Docker, Kubernetes, eBPF, observability, CI/CD, CLI tool, build system, Vercel, container
P2	Startups & Builders	YC, shipped, launched, users, milestone, open source, founder, builder, directory
P2	Science & Space	SpaceX, rocket, Starship, physics, quantum, space
P2	OSS & PKM	Obsidian, second brain, knowledge, awesome-list, PKM, open-source
P3	Everything else	Sort last, still show

Classification Rules

1. Scan text + author for keyword matches
2. Assign highest matching tier (P0 wins over P1)
3. Within each tier, sort by engagement (likes + retweets + points + stars)
4. Remove exact duplicates and RTs of already-shown items
5. Extract links from P0 items as action items

Key Accounts (always surface when they appear)

Customization: Replace with accounts YOU follow that are high-signal for your interests.

# AI / LLM Engineering
@hwchase17 @AnthropicAI @karpathy @swyx @jxnlco @simonw @tom_doerr

# Systems / Infra / Tooling
@rauchg @cramforce @mitchellh @kelseyhightower @ThePrimeagen @antirez

# AI Research / Models
@_lewtun @vllm_project @huggingface @GoogleDeepMind

# Hardware / Science
@elonmusk @dfrobotcn @LensScientific

# Builders
@kepano @DanielleFong @levelsio @zenorocha

Phase 3: DISPLAY — TUI Cards

ALL content rendered as cards. Never plain tables or bullets for content items.

Tweet Card

┌──────────────────────────────────────────────────────────────────────────┐
│ ○ #N  @author                                        [P0] Agentic AI    │
│       ♥ 10.7k   ↺ 723   ◎ 184          Apr 12, 2026 · 3h ago          │
│                                                                          │
│  Tweet text wrapped at ~70 chars, max 4 lines                            │
│  Truncate with … if longer                                               │
│                                                                          │
│  → https://x.com/i/web/status/ID                                        │
└──────────────────────────────────────────────────────────────────────────┘

HN Card

┌──────────────────────────────────────────────────────────────────────────┐
│ ○ #N  source: Hacker News                            [P1] Databases     │
│       ▲ 203 points   💬 54 comments     Apr 11, 2026 · 1d ago          │
│                                                                          │
│  Show HN: Postgres extension for BM25 full-text search                  │
│                                                                          │
│  → https://github.com/timescale/pg_textsearch                            │
└──────────────────────────────────────────────────────────────────────────┘

GitHub Card

┌──────────────────────────────────────────────────────────────────────────┐
│ ○ #N  source: GitHub                                 [P1] Infra         │
│       ⭐ 1.2k stars   🍴 89 forks   lang: Go        Apr 10, 2026 · 2d ago │
│                                                                          │
│  owner/repo — Description text here                                      │
│                                                                          │
│  → https://github.com/owner/repo                                         │
└──────────────────────────────────────────────────────────────────────────┘

arXiv Card

┌──────────────────────────────────────────────────────────────────────────┐
│ ○ #N  source: arXiv                                  [P0] ML Research   │
│       📄 cs.AI                                  published: Apr 11, 2026  │
│                                                                          │
│  Paper Title Here                                                        │
│  Authors: First, Second, Third                                           │
│                                                                          │
│  → https://arxiv.org/abs/2604.12345                                      │
└──────────────────────────────────────────────────────────────────────────┘

HuggingFace Card

┌──────────────────────────────────────────────────────────────────────────┐
│ ○ #N  source: HuggingFace                            [P0] Models        │
│       ❤️ 542 likes   ⬇ 12.4k downloads   task: text-generation         │
│                                                                          │
│  org/model-name — 7B param, Apache 2.0                                  │
│                                                                          │
│  → https://huggingface.co/org/model-name                                 │
└──────────────────────────────────────────────────────────────────────────┘

Lobsters / Reddit Card

┌──────────────────────────────────────────────────────────────────────────┐
│ ○ #N  source: Lobste.rs                              [P1] Systems       │
│       ▲ 45 score   🏷 rust, cli             Apr 12, 2026 · 5h ago      │
│                                                                          │
│  Title of the post                                                       │
│                                                                          │
│  → https://lobste.rs/s/abc123                                            │
└──────────────────────────────────────────────────────────────────────────┘

Card Rules

1. Box width: 76 inner (78 with border). Fixed.
2. Header: read-status icon + source/author left, [tier] topic right-aligned
3. Date line: engagement stats LEFT, date + relative time RIGHT (e.g., Apr 12, 2026 · 3h ago)
4. Engagement: K/M suffixes for >999
5. Text: wrap ~70 chars, max 4 lines, truncate with …
6. Link: always → https://FULL_URL at bottom (MUST include https:// prefix so terminals render clickable links)
7. Tier section headers: ━━━ P0: AGENTIC AI ━━━━━━━━━━━━━━━━━━━━━━━━━━━

Date Display Rules

• Twitter: use the time field from JSON, convert to MMM DD, YYYY · Nh ago or Nd ago
• HN: use created_at field
• GitHub: use created_at field from repo
• Lobsters: use created_at field
• HuggingFace: use lastModified field
• arXiv: use published field
• Relative time: Xm ago if <1h, Xh ago if <24h, Xd ago if <30d, just date if older

Read Status Icons

• ○ = unread (not yet read/actioned)
• ● = read (seen/actioned)

Default: all items start as ○ (unread). When user says "mark read", "done", "read #N", or discusses a specific item in detail, flip it to ● in the reading list.

Action Items Box (after all cards)

┌─── ACTION ITEMS ──────────────────────────────────────────────────────┐
│  📄 @author — description → https://URL                                │
│  ⭐ repo — description → https://URL                                   │
│  🔧 tool — description → https://URL                                   │
└──────────────────────────────────────────────────────────────────────┘

Summary Footer

━━━ 42 items | P0: 10 · P1: 6 · P2: 8 · P3: 18 | ○ 38 unread · ● 4 read | 3 action items ━━━

Phase 3.5: READING LIST — Track Read Status

After displaying cards, save/update a reading list at ~/.secondmem/reading-list.md.

Reading List File Format

# Reading List

## Fetched: 2026-04-12 22:30

### Unread (○)
| # | Status | Tier | Source | Author/Repo | Summary | URL | Date |
|---|--------|------|--------|-------------|---------|-----|------|
| 1 | ○ | P0 | GitHub | hermes-agent-orange-book | Agent framework guide | github.com/... | Apr 10 |
| 2 | ○ | P0 | Twitter | @sharbel | AI agents hijacked by websites | x.com/... | Apr 12 |

### Read (●)
| # | Status | Tier | Source | Author/Repo | Summary | URL | Date | Read On |
|---|--------|------|--------|-------------|---------|-----|------|---------|
| 4 | ● | P0 | Twitter | @tom_doerr | 1000+ agent skills | x.com/... | Apr 12 | Apr 12 |

Reading List Rules

1. On every fetch: Append new items to Unread (○) section. Don't duplicate (match by URL).
2. Mark as read: "mark #N read", "done with #N", "read #N" → move from Unread to Read, add Read On date.
3. Mark batch: "mark all P0 read", "done with [topic]" → move matching items.
4. Show unread: "what's pending", "unread items" → display ○ items as cards.
5. Show read: "what did I read", "reading history" → show ● items.
6. Auto-mark: When user discusses a specific item in depth, mark it ●.
7. Cleanup: Items older than 30 days in Read section → archive to reading-list-archive.md.

Phase 4: INGEST into Knowledge Base

After displaying cards, automatically ingest P0 and P1 items into ~/.secondmem/knowledge/.

Knowledge base structure: Topic-first directories with month-scoped content files. Creates the directory structure automatically on first run.

Domain → Directory Mapping

Tier/Domain	Target Directory	File Strategy
Agentic AI	/ai-ml/	Create or upsert agentic-ai-YYYY-MMM.md
Foundational ML	/ai-ml/	Create or upsert ml-models-YYYY-MMM.md
GPU / Hardware	/engineering/	Create or upsert gpu-hardware-landscape.md
Databases / System Design	/engineering/	Create or upsert database-internals-YYYY-MMM.md
Infra	/engineering/	Create or upsert into relevant existing file
Startups	/startups/	Create or upsert based on sub-topic
Papers	/research/	One file per significant paper
People insights	/people-insights/	Group by person
Frontend	/engineering/	Create or upsert frontend-YYYY-MMM.md
Security	/engineering/	Create or upsert security-YYYY-MMM.md

Ingest Rules

1. Read target directory's hierarchy.md first (create if missing)
2. Check if relevant file exists — upsert if yes, create if no
3. Use month-scoped filenames for rolling content (database-internals-2026-apr.md)
4. Write content in structured markdown format (Topic, Source, Ingested, Tags, sections)
5. Tweet format: > "text" — @author, date + "Why this matters" + extracted links
6. Repo format: name, stars, language, description, why it matters
7. Paper format: title, authors, key claims, link
8. Every item MUST carry per-item dates in the source blockquote:

   > **Posted:** YYYY-MM-DD · **Ingested:** YYYY-MM-DD
   

   • Posted = original publish/tweet date (from API time, created_at, or page date)
   • Ingested = today's date when written to knowledge base
   • Place immediately after the engagement line in the blockquote
9. Update directory hierarchy.md after writes
10. Update root hierarchy.md if new files created
11. Update ~/.secondmem/timeline.md — append new items to current month section
12. Log to ~/.secondmem/logs/ingest.log
13. Max 1116 lines per file — split if exceeded
14. Cross-reference new files with related existing files (3-8 refs)

Date Organization Strategy

• Primary axis: topic-first — files live in topic directories (/ai-ml/, /engineering/)
• Secondary axis: month-scoped filenames — rolling content uses topic-YYYY-mmm.md
• Chronological index: ~/.secondmem/timeline.md — reverse-chronological index for date-based recall
• "What did I learn last week?" → read timeline.md
• "What do I know about databases?" → read topic directory
• Both paths lead to the same content files — timeline is an index, not a copy

Timeline Index Format (~/.secondmem/timeline.md)

## 2026-04 (April)

| Date | Topic | Title | File | Source |
|------|-------|-------|------|--------|
| Apr 12 | Agentic AI | Agent web hijacking | ai-ml/agentic-ai-2026-apr.md | @sharbel |
| Apr 11 | Databases | Postgres BM25 extension | engineering/database-internals-2026-apr.md | timescale |

• Grouped by month, reverse chronological within each month
• One row per ingested item (not per file)
• Keep last 6 months; archive older to timeline-archive-YYYY.md

What gets ingested vs skipped

• Always ingest: P0 items, P1 items with significant engagement (>50 points/likes)
• Selectively ingest: P2 items that are repos, papers, or tool launches
• Skip: P3 items, engagement bait, generic motivation, duplicate RTs
• Always extract: Links from P0 tweets → separate entries for papers/repos

Knowledge Base Init (first run)

If ~/.secondmem/knowledge/hierarchy.md doesn't exist, create the base structure:

mkdir -p ~/.secondmem/knowledge/{ai-ml,engineering,startups,research,people-insights}
mkdir -p ~/.secondmem/logs
# Create root hierarchy.md and per-directory hierarchy.md files

Phase 5: CONFIRM

After all phases complete, show summary:

━━━ mo-feed complete ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Fetched:  Twitter (50) + HN (10) + GitHub (10) + Lobsters (10) = 80 items
Unique:   62 after dedup
Classified: P0: 12 · P1: 18 · P2: 14 · P3: 18
Ingested: 24 items into knowledge base (12 P0 + 12 P1)
Files:    engineering/database-internals-2026-apr.md (updated)
          ai-ml/agentic-ai-2026-apr.md (created)
Reading:  24 new items added to reading list (○ unread)
Action:   5 links to check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Modes

Mode	Trigger	Behavior
Full scan	"catch me up", "daily digest", "mo feed"	All sources, all tiers, full ingest
Topic scan	"what's new in [topic]"	Topic-specific Twitter search + matching cross-refs
Source scan	"scan twitter", "scan HN"	Single source, all tiers, ingest P0/P1
Quick peek	"quick feed", "headlines"	Twitter timeline only, cards only, no ingest
Query KB	"what do I know about [topic]"	Read from knowledge base, no fetch
Ingest only	"ingest this", "remember this"	No fetch, write provided content to KB
Show unread	"what's pending", "unread"	Show ○ items from reading-list.md as cards
Show read	"what did I read", "reading history"	Show ● items from reading-list.md
Mark read	"read #N", "done with #N"	Move item from ○ to ● in reading-list.md
Mark batch	"mark all P0 read", "done with [topic]"	Move matching tier/topic items to ●

Agent Instructions

1. ALWAYS fetch with --json for Twitter (for classification)
2. Run cross-reference fetches in PARALLEL (independent curl calls)
3. Classify ALL items before displaying any cards
4. Display cards grouped by tier, sorted by engagement within tier
5. After displaying: ingest P0 + high-engagement P1 into knowledge base
6. Every ingested item MUST have > **Posted:** YYYY-MM-DD · **Ingested:** YYYY-MM-DD
7. Update hierarchy.md files after ingest
8. Append new items to ~/.secondmem/timeline.md under the current month section
9. Log all operations to ~/.secondmem/logs/ingest.log
10. Save catch-up file to ~/Documents/mo-feed-catchup-YYYY-MM-DD.md for >20 items
11. Default to --lang en for Twitter
12. If a fetch fails (timeout, no Chrome), skip that source and continue with others
13. Never block the whole pipeline on one source failure
14. On first run, create ~/.secondmem/ directory structure if it doesn't exist

Customization Guide

Adding Your Own Interest Tiers

Edit the Priority Tiers table in Phase 2. Add keywords that matter to you:

| **P0** | Your Domain | keyword1, keyword2, keyword3 |

Adding Your Key Accounts

Replace the Key Accounts list with Twitter handles you want to always surface.

Adding New Sources

Add a new entry to the Source-specific fetch table with the API/curl command. The classifier works on any text content — just feed it JSON with a text or title field.

Changing the Knowledge Base Path

The default knowledge base path is ~/.secondmem/knowledge/. To change it, update all references to ~/.secondmem/ in this file.

Disabling Ingest

For a read-only feed (no knowledge base writes), use Quick peek mode: say "quick feed" or "headlines" instead of "catch me up".