fetch-tweets

name	fetch-tweets
description	Search and curate X/Twitter behind one selector - keyword, topic roundup, a single or tracked-account digest, an X list, or the AI-agent buzz preset - clustered into signal-scored sub-narratives.
metadata	{"title":"Fetch Tweets","category":"basics","var":"","tags":["social"],"requires":["XAI_API_KEY?"]}

${var} — <source>:<arg> where <source> ∈ keyword | topic | account | list | agent-buzz. The <arg> is source-specific (a query, a topic, a handle, comma-separated list IDs, or an optional focus). If no source: prefix is given, the source is inferred from the shape of <arg> (see Source selector). Required for keyword and list; optional for topic, account, and agent-buzz.

Today is ${today}. This skill fetches X/Twitter content along one of five source axes and produces a curated digest — clustered by sub-narrative, ranked by signal, one insight per item — never a flat chronological dump.

Source selector

Parse ${var} into SOURCE and ARG before doing anything else.

Explicit form (recommended): <source>:<arg>

• keyword:$SOL OR solana OR "solana network" — raw X search query, passed to Grok verbatim (OR/AND honored).
• topic:brain-computer interfaces — a single topic roundup. topic: (empty arg) → resolve a topic list from MEMORY.md, then built-in defaults.
• account:vitalikbuterin — one account's recent tweets. account: (empty arg) → digest every handle in memory/topics/tracked-accounts.yml.
• list:1953536336675365173,1937207796270829766 — one or more numeric X list IDs. Append |<topic> for a topic booster: list:195...,193...|AI agents.
• agent-buzz — the curated AI-agent-ecosystem preset. agent-buzz:MCP protocol prioritizes a project/topic within the preset.

Implicit form (back-compat with migrated bare-var configs): when ${var} has no recognized source: prefix, infer SOURCE in this order:

1. ${var} is empty → topic (default multi-topic roundup).
2. ${var} is all-digits, or comma-separated all-digits (optionally with a |<topic> suffix) → list.
3. ${var} is @handle or matches ^[A-Za-z0-9_]{1,15}$ (a bare handle) → account.
4. Anything else → keyword.

Note: agent-buzz has no distinct implicit shape (its arg looks like a keyword/topic), so it is only selectable via the explicit agent-buzz / agent-buzz:... prefix.

Once SOURCE and ARG are set, jump to the matching branch below. Only one branch runs per invocation.

Shared preamble (all branches)

1. Read memory/MEMORY.md for context and the recent memory/logs/ (each branch specifies its lookback window — 2 or 3 days) to dedup already-reported tweets.

2. Load the dedup set SEEN_TWEETS by unioning two sources:

   • The branch's persistent seen-file (per-mode path below), if it exists — read all URLs.
   • The branch's log lookback window — grep each memory/logs/*.md file in range for lines matching https://x.com/.

   Per-mode seen-files (kept at their legacy paths so dedup history survives the merge):

   mode	seen-file	log lookback
   keyword	memory/fetch-tweets-seen.txt	3 days
   topic	memory/tweet-roundup-seen.txt	3 days
   account	(logs only — see branch)	2 days
   list	memory/list-digest-seen.txt	2 days
   agent-buzz	(logs only — 3-day status/<id> set)	3 days

3. Formatting invariants shared by every branch's notification:

   • Use x.com/handle (never @handle) so Telegram doesn't ping/tag users. (Exception: the account-digest and agent-buzz formats below historically use @handle in-body; keep their documented format but prefer x.com/handle when practical.)
   • Every surviving tweet gets a tappable Markdown link — [View](url) / [View tweet](url). If a URL is unavailable, drop the link and say "(link unavailable)".
   • Never fabricate engagement counts. Missing → 0, not a guess.
   • Notify only on signal. A legitimately empty or all-duplicate run logs its status and sends nothing.

Voice

Used by the account and agent-buzz branches for one-line takes/insights. If soul/SOUL.md and soul/STYLE.md are populated, read both and match the operator's voice. If they are empty templates or absent, write in a clear, direct, neutral tone — state what the tweet says, no hedging or editorializing beyond the tweet itself.

---

Branch: keyword (source:keyword)

Search X for tweets matching ARG and produce a curated digest grouped by sub-narrative.

Seen set: memory/fetch-tweets-seen.txt + last 3 days of logs (loaded in preamble).

1. Build the search prompt. Pass ARG to Grok verbatim as the query — do NOT narrow it to a single angle; broad coverage is the goal. Ask for at least 15–20 candidate tweets (you'll cull to ~7–10). Always require explicit engagement counts (likes, retweets, replies) so ranking is data-driven.

2. Fetch tweets. Record SOURCE_PATH=api|websearch for the log.

   Path A — X.AI API (primary; see the Fetching (all branches) contract — attempt this, set the Bash tool timeout ≥180000, capture the HTTP status):

   FROM_DATE=$(date -u -d "yesterday" +%Y-%m-%d 2>/dev/null || date -u -v-1d +%Y-%m-%d)
   TO_DATE=$(date -u +%Y-%m-%d)
   PROMPT="Search X for tweets about: ${ARG}. Date range: ${FROM_DATE} to ${TO_DATE}. Return at least 15-20 candidate tweets — mix of high-engagement posts and smaller accounts that add a distinct angle. For each tweet include: @handle, the full text, date posted, exact engagement counts (likes, retweets, replies — never N/A; if unknown, say 0), and the direct link (https://x.com/handle/status/ID). Return as a numbered list."
   jq -n --arg p "$PROMPT" '{model:"grok-4.6", input:[{role:"user",content:$p}], tools:[{type:"x_search"}]}' > /tmp/xai-ft-keyword.json
   HTTP=$(./secretcurl -s -o /tmp/xai.json -w '%{http_code}' --max-time 150 -X POST "https://api.x.ai/v1/responses" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer {XAI_API_KEY}" \
     -d @/tmp/xai-ft-keyword.json)
   echo "xai http=$HTTP bytes=$(wc -c </tmp/xai.json)"
   

   On HTTP=200, parse /tmp/xai.json with: jq -r '.output[] | select(.type == "message") | .content[] | select(.type == "output_text") | .text' and mark SOURCE_PATH=api.

   Path B — WebSearch fallback (only if the key is KEY_UNSET, or Path A gave a non-2xx / empty / timeout per the contract): use the built-in WebSearch tool with site:x.com "<query terms>" after:${FROM_DATE}. Note at the top of the log the true reason (http-<code> / timeout / empty, never "unavailable" when the key was set) and "results compiled via WebSearch — quality lower than usual". WebSearch favours high-engagement older tweets — prioritise results dated within the last 48 hours. Mark SOURCE_PATH=websearch.

3. Empty vs. error handling (distinguish):

   • Legitimate empty (0 tweets): log FETCH_TWEETS_EMPTY (source=${SOURCE_PATH}) and stop — no notification.
   • API/cache error (HTTP error, malformed JSON, all paths failed): log FETCH_TWEETS_ERROR (last_path=${SOURCE_PATH}, reason=...) and stop — no notification.

4. Deduplicate each candidate URL against SEEN_TWEETS. If ALL are dupes: log FETCH_TWEETS_NO_NEW: all results already reported and stop — no notification.

5. Curate (the core step): a. Cluster survivors into 2–4 sub-narratives by what they're claiming/discussing (e.g. for a token: "price action", "team announcement", "criticism/FUD", "ecosystem integration"). Name the angle, not the topic. b. Rank within each cluster by signal (not raw engagement): signal = likes + 2×retweets + replies, but demote pure replies, generic shilling, and near-duplicate paraphrases. Drop tweets with <5 total engagement unless they add a unique angle. c. Cap each cluster at 2–3 tweets, total 7–10. Quality over quantity — if only 5 pass, send 5. Don't pad. d. Extract the claim/signal per tweet — what's new or interesting, not a literal paraphrase. Bad: "User says token is going up." Good: "Calls out the team's silence on the postponed unlock — first major holder to do so publicly." e. Compute a one-line signal for the top of the notification — one observation about the shape of the conversation (e.g. "Sentiment split — 4 bullish on the launch, 3 critical of the unlock terms.").

6. Save + update seen-file (see Log). Append each kept tweet URL (one per line) to memory/fetch-tweets-seen.txt (create if missing).

7. Notify via ./notify with the clustered output:

   *Top Tweets — ${ARG} (${today})*
   _${signal_one_liner}_
   
   *${cluster_1_name}*
   1. x.com/handle — [insight summary]
   Likes: X | RTs: Y | Replies: Z
   [View tweet](https://x.com/handle/status/ID)
   
   2. x.com/handle — [insight summary]
   Likes: X | RTs: Y | Replies: Z
   [View tweet](https://x.com/handle/status/ID)
   
   *${cluster_2_name}*
   3. x.com/handle — [insight summary]
   ...
   

   The signal one-liner is italic (_..._) directly under the title; cluster headers are *bold*.

Status codes: FETCH_TWEETS_OK (notified) | FETCH_TWEETS_EMPTY | FETCH_TWEETS_ERROR | FETCH_TWEETS_NO_NEW.

---

Branch: topic (source:topic)

Gist of the latest X chatter on one or more configurable topics.

Seen set: memory/tweet-roundup-seen.txt + last 3 days of logs.

1. Resolve the topic list (priority order):

   1. ARG set → TOPICS=("$ARG") (single-topic mode).
   2. Else if MEMORY.md has a ## Tweet Roundup Topics section → use its bulleted lines, one query per line.
   3. Else built-in defaults:
      • artificial intelligence OR AI agents OR LLM
      • crypto OR bitcoin OR DeFi
      • technology OR startups OR open source

2. Fetch per topic — track SOURCE ∈ {api, websearch, failed} per topic.

   Path A — direct X.AI curl (primary): for each topic, call Grok's x_search.

   FROM_DATE=$(date -u -d "yesterday" +%Y-%m-%d 2>/dev/null || date -u -v-1d +%Y-%m-%d)
   TO_DATE=$(date -u +%Y-%m-%d)
   PROMPT="Search X for recent tweets about: ${TOPIC}. Date range: ${FROM_DATE} to ${TO_DATE}. Return up to 8 substantive tweets. For each: @handle, full text, date, exact engagement counts (likes, retweets, replies; 0 if unknown), and the direct link https://x.com/handle/status/ID."
   jq -n --arg p "$PROMPT" '{model:"grok-4.6", input:[{role:"user",content:$p}], tools:[{type:"x_search"}]}' > /tmp/xai-ft-topic.json
   ./secretcurl -s -o /tmp/xai-topic-out.json -X POST "https://api.x.ai/v1/responses" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer {XAI_API_KEY}" \
     -d @/tmp/xai-ft-topic.json
   

   Parse with the standard jq extractor. If it yields text, SOURCE=api. Extract each tweet's @handle, text, engagement counts, and permalink.

   Path B — WebSearch fallback (only if XAI_API_KEY unset, or Path A errors/empty): site:x.com "<topic keywords>" after:<YESTERDAY>. Always include the word "today" and ${today} to force fresh results. Discard any result whose visible date is older than 48h. Collect up to 5 candidates per topic. Mark SOURCE=websearch. If both paths return nothing, mark SOURCE=failed.

3. Score and filter. Require: a known @handle; a https://x.com/<handle>/status/<id> URL (if missing, keep but mark "link unavailable"); posted within 48h; URL not in SEEN_TWEETS. Compute signal_score = likes + 2×retweets + replies (on WebSearch path with no counts, use result rank as a weak proxy). Demote −50%: replies to a parent tweet; near-duplicates of a higher-scoring tweet (>70% text overlap or same linked URL).

4. Curate per topic:

   • 0 survivors → drop the topic. Do NOT pad.
   • 1–3 survivors → list ranked by signal_score, highest first.
   • 4+ survivors → group into 2–3 sub-narratives (shared keywords/entity/claim); label each, surface the top-1 tweet per narrative as exemplar. Write an insight per reported tweet (what it asserts/reveals, not a headline paraphrase). Write a one-line conversation shape per topic ("bullish momentum, dissenters quiet", "split opinion on X's launch", "single story dominating — Y").

5. Notify. If every topic dropped: log TWEET_ROUNDUP_EMPTY and stop — no notify. Otherwise send via ./notify (≤4000 chars):