fetch-tweets

<!-- autoresearch: variation B — sharper output via clustering + signal scoring + insight extraction. Merged HUB: absorbs tweet-digest, tweet-roundup, list-digest, refresh-x, agent-buzz behind a `source:` selector. --> > **${var}** — `<source>:<arg>` where `<source>` ∈ `keyword | topic | account | list | agent-buzz`. The `<arg>` is source-specific (a query, a topic, a handle, comma-separated list IDs, or an optional focus). If no `source:` prefix is given, the source is inferred from the shape of `<arg>` (see **Source selector**). **Required** for `keyword` and `list`; optional for `topic`, `account`, and `agent-buzz`. Today is ${today}. This skill fetches X/Twitter content along one of five **source axes** and produces a *curated* digest — clustered by sub-narrative, ranked by signal, one insight per item — never a flat chronological dump. ## Source selector Parse `${var}` into `SOURCE` and `ARG` before doing anything else. **Explicit form (recommended):** `<source>:<arg>` - `keyword:$SOL OR solana OR "solana network"` — raw X search query, passed to Grok **verbatim** (OR/AND honored). - `topic:brain-computer interfaces` — a single topic roundup. `topic:` (empty arg) → resolve a topic **list** from MEMORY.md, then built-in defaults. - `account:vitalikbuterin` — one account's recent tweets. `account:` (empty arg) → digest **every** handle in `memory/topics/tracked-accounts.yml`. - `list:1953536336675365173,1937207796270829766` — one or more numeric X list IDs. Append `|<topic>` for a topic booster: `list:195...,193...|AI agents`. - `agent-buzz` — the curated AI-agent-ecosystem preset. `agent-buzz:MCP protocol` prioritizes a project/topic within the preset. **Implicit form (back-compat with migrated bare-var configs):** when `${var}` has **no** recognized `source:` prefix, infer `SOURCE` in this order: 1. `${var}` is empty → `topic` (default multi-topic roundup). 2. `${var}` is all-digits, or comma-separated all-digits (optionally with a `|<topic>` suffix) → `list`. 3. `${var}` is `@handle` or matches `^[A-Za-z0-9_]{1,15}$` (a bare handle) → `account`. 4. Anything else → `keyword`. Note: `agent-buzz` has **no** distinct implicit shape (its arg looks like a keyword/topic), so it is **only** selectable via the explicit `agent-buzz` / `agent-buzz:...` prefix. Once `SOURCE` and `ARG` are set, jump to the matching branch below. Only one branch runs per invocation. ## Shared preamble (all branches) 1. Read `memory/MEMORY.md` for context and the recent `memory/logs/` (each branch specifies its lookback window — 2 or 3 days) to dedup already-reported tweets. 2. **Load the dedup set `SEEN_TWEETS`** by unioning two sources: - The branch's **persistent seen-file** (per-mode path below), if it exists — read all URLs. - The branch's **log lookback window** — grep each `memory/logs/*.md` file in range for lines matching `https://x.com/`. Per-mode seen-files (kept at their legacy paths so dedup history survives the merge): | mode | seen-file | log lookback | |---|---|---| | keyword | `memory/fetch-tweets-seen.txt` | 3 days | | topic | `memory/tweet-roundup-seen.txt` | 3 days | | account | *(logs only — see branch)* | 2 days | | list | `memory/list-digest-seen.txt` | 2 days | | agent-buzz | *(logs only — 3-day `status/<id>` set)* | 3 days | 3. Formatting invariants shared by **every** branch's notification: - Use `x.com/handle` (**never** `@handle`) so Telegram doesn't ping/tag users. *(Exception: the account-digest and agent-buzz formats below historically use `@handle` in-body; keep their documented format but prefer `x.com/handle` when practical.)* - Every surviving tweet gets a tappable Markdown link — `[View](url)` / `[View tweet](url)`. If a URL is unavailable, drop the link and say "(link unavailable)". - Never fabricate engagement counts. Missing → `0`, not a guess. - **Notify only on signal.** A legitimately empty or all-duplicate run logs its status and sends **nothing**. ## Voice Used by the `account` and `agent-buzz` branches for one-line takes/insights. If `soul/SOUL.md` and `soul/STYLE.md` are populated, read both and match the operator's voice. If they are empty templates or absent, write in a clear, direct, neutral tone — state what the tweet says, no hedging or editorializing beyond the tweet itself. --- ## Branch: keyword (`source:keyword`) Search X for tweets matching `ARG` and produce a curated digest grouped by sub-narrative. **Seen set:** `memory/fetch-tweets-seen.txt` + last 3 days of logs (loaded in preamble). 1. **Build the search prompt.** Pass `ARG` to Grok **verbatim** as the query — do NOT narrow it to a single angle; broad coverage is the goal. Ask for **at least 15–20 candidate tweets** (you'll cull to ~7–10). Always require explicit engagement counts (likes, retweets, replies) so ranking is data-driven. 2. **Fetch tweets.** Record `SOURCE_PATH=api|websearch` for the log. **Path A — X.AI API** (primary; see the **Fetching (all branches)** contract — attempt this, set the Bash tool `timeout` ≥180000, capture the HTTP status): ```bash FROM_DATE=$(date -u -d "yesterday" +%Y-%m-%d 2>/dev/null || date -u -v-1d +%Y-%m-%d) TO_DATE=$(date -u +%Y-%m-%d) PROMPT="Search X for tweets about: ${ARG}. Date range: ${FROM_DATE} to ${TO_DATE}. Return at least 15-20 candidate tweets — mix of high-engagement posts and smaller accounts that add a distinct angle. For each tweet include: @handle, the full text, date posted, exact engagement counts (likes, retweets, replies — never N/A; if unknown, say 0), and the direct link (https://x.com/handle/status/ID). Return as a numbered list." jq -n --arg p "$PROMPT" '{model:"grok-4.6", input:[{role:"user",content:$p}], tools:[{type:"x_search"}]}' > /tmp/xai-ft-keyword.json HTTP=$(./secretcurl -s -o /tmp/xai.json -w '%{http_code}' --max-time 150 -X POST "https://api.x.ai/v1/responses" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {XAI_API_KEY}" \ -d @/tmp/xai-ft-keyword.json) echo "xai http=$HTTP bytes=$(wc -c </tmp/xai.json)" ``` On `HTTP=200`, parse `/tmp/xai.json` with: `jq -r '.output[] | select(.type == "message") | .content[] | select(.type == "output_text") | .text'` and mark `SOURCE_PATH=api`. **Path B — WebSearch fallback** (only if the key is `KEY_UNSET`, or Path A gave a non-2xx / empty / timeout per the contract): use the built-in WebSearch tool with `site:x.com "<query terms>" after:${FROM_DATE}`. Note at the top of the log the **true reason** (`http-<code>` / `timeout` / `empty`, never "unavailable" when the key was set) and "results compiled via WebSearch — quality lower than usual". WebSearch favours high-engagement older tweets — **prioritise results dated within the last 48 hours**. Mark `SOURCE_PATH=websearch`. 3. **Empty vs. error handling** (distinguish): - **Legitimate empty** (0 tweets): log `FETCH_TWEETS_EMPTY (source=${SOURCE_PATH})` and **stop — no notification**. - **API/cache error** (HTTP error, malformed JSON, all paths failed): log `FETCH_TWEETS_ERROR (last_path=${SOURCE_PATH}, reason=...)` and **stop — no notification**. 4. **Deduplicate** each candidate URL against `SEEN_TWEETS`. If ALL are dupes: log `FETCH_TWEETS_NO_NEW: all results already reported` and **stop — no notification**. 5. **Curate** (the core step): a. **Cluster** survivors into 2–4 sub-narratives by what they're claiming/discussing (e.g. for a token: "price action", "team announcement", "criticism/FUD", "ecosystem integration"). Name the *angle*, not the topic. b. **Rank within each cluster by signal** (not raw engagement): `signal = likes + 2×retweets + replies`, but **demote** pure replies, generic shilling, and near-duplicate paraphrases. Drop tweets with <5 total engagement unless they add a unique angle. c. **Cap each cluster at 2–3 tweets, total 7–10.** Quality over quantity — if only 5 pass, send 5. Don't pad. d. **Extract the claim/signal** per tweet — *what's new or interesting*, not a literal paraphrase. Bad: "User says token is going up." Good: "Calls out the team's silence on the postponed unlock — first major holder to do so publicly." e. **Compute a one-line signal** for the top of the notification — one observation about the *shape* of the conversation (e.g. "Sentiment split — 4 bullish on the launch, 3 critical of the unlock terms."). 6. **Save + update seen-file** (see Log). Append each kept tweet URL (one per line) to `memory/fetch-tweets-seen.txt` (create if missing). 7. **Notify via `./notify`** with the clustered output: ``` *Top Tweets — ${ARG} (${today})* _${signal_one_liner}_ *${cluster_1_name}* 1. x.com/handle — [insight summary] Likes: X | RTs: Y | Replies: Z [View tweet](https://x.com/handle/status/ID) 2. x.com/handle — [insight summary] Likes: X | RTs: Y | Replies: Z [View tweet](https://x.com/handle/status/ID) *${cluster_2_name}* 3. x.com/handle — [insight summary] ... ``` The signal one-liner is italic (`_..._`) directly under the title; cluster headers are `*bold*`. **Status codes:** `FETCH_TWEETS_OK` (notified) | `FETCH_TWEETS_EMPTY` | `FETCH_TWEETS_ERROR` | `FETCH_TWEETS_NO_NEW`. --- ## Branch: topic (`source:topic`) Gist of the latest X chatter on one or more configurable topics. **Seen set:** `memory/tweet-roundup-seen.txt` + last 3 days of logs. 1. **Resolve the topic list** (priority order): 1. `ARG` set → `TOPICS=("$ARG")` (single-topic mode). 2. Else if MEMORY.md has a `## Tweet Roundup Topics` section → use its bulleted lines, one query per line. 3. Else built-in defaults: - `artificial intelligence OR AI agents OR LLM` - `crypto OR bitcoin OR DeFi` - `technology OR startups OR open source` 2. **Fetch per topic** — track `SOURCE ∈ {api, websearch, failed}` per topic. **Path A — direct X.AI curl** (primary): for each topic, call Grok's `x_search`. ```bash FROM_DATE=$(date -u -d "yesterday" +%Y-%m-%d 2>/dev/null || date -u -v-1d +%Y-%m-%d) TO_DATE=$(date -u +%Y-%m-%d) PROMPT="Search X for recent tweets about: ${TOPIC}. Date range: ${FROM_DATE} to ${TO_DATE}. Return up to 8 substantive tweets. For each: @handle, full text, date, exact engagement counts (likes, retweets, replies; 0 if unknown), and the direct link https://x.com/handle/status/ID." jq -n --arg p "$PROMPT" '{model:"grok-4.6", input:[{role:"user",content:$p}], tools:[{type:"x_search"}]}' > /tmp/xai-ft-topic.json ./secretcurl -s -o /tmp/xai-topic-out.json -X POST "https://api.x.ai/v1/responses" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer {XAI_API_KEY}" \ -d @/tmp/xai-ft-topic.json ``` Parse with the standard `jq` extractor. If it yields text, `SOURCE=api`. Extract each tweet's `@handle`, text, engagement counts, and permalink. **Path B — WebSearch fallback** (only if `XAI_API_KEY` unset, or Path A errors/empty): `site:x.com "<topic keywords>" after:<YESTERDAY>`. Always include the word "today" and `${today}` to force fresh results. Discard any result whose visible date is older than 48h. Collect up to 5 candidates per topic. Mark `SOURCE=websearch`. If both paths return nothing, mark `SOURCE=failed`. 3. **Score and filter.** Require: a known `@handle`; a `https://x.com/<handle>/status/<id>` URL (if missing, keep but mark "link unavailable"); posted within 48h; URL **not** in `SEEN_TWEETS`. Compute `signal_score = likes + 2×retweets + replies` (on WebSearch path with no counts, use result rank as a weak proxy). **Demote −50%**: replies to a parent tweet; near-duplicates of a higher-scoring tweet (>70% text overlap or same linked URL). 4. **Curate per topic:** - **0 survivors** → drop the topic. Do NOT pad. - **1–3 survivors** → list ranked by `signal_score`, highest first. - **4+ survivors** → group into 2–3 sub-narratives (shared keywords/entity/claim); label each, surface the top-1 tweet per narrative as exemplar. Write an **insight** per reported tweet (what it asserts/reveals, not a headline paraphrase). Write a one-line **conversation shape** per topic ("bullish momentum, dissenters quiet", "split opinion on X's launch", "single story dominating — Y"). 5. **Notify.** If every topic dropped: log `TWEET_ROUNDUP_EMPTY` and **stop — no notify**. Otherwise send via `./notify` (≤4000 chars): ```