x-tweet-by-conversation

name

x-tweet-by-conversation

description

Collects every tweet in an X (Twitter) conversation thread given a conversation id (root tweet id) — the focal tweet plus all replies, sub-replies, and quote chains — and returns normalized per-tweet data with text, author, engagement counts, media, hashtags, mentions, in_reply_to mapping, and cursor for pagination. Use when user mentions Twitter conversation, X conversation thread, conversation_id, thread scraper, scrape Twitter replies, all replies to a tweet, replies under a tweet, sub-replies, nested replies, thread harvester, get replies of a tweet, scrape comments on Twitter, scrape comments on X, full thread extraction, conversation export, conversation tree, reply chain, thread dump, X tweet thread, twitter thread scrape, comment scraping twitter, comment scraping x, focal tweet plus context, root tweet plus replies. Also applies to sentiment analysis on a single viral tweet, controversy mapping, harvesting community Q&A threads, capturing AMA threads, recovering long-running discussions, and any paginated bulk reply collection driven by a conversation id.

X — Tweets by Conversation

Conversation id (root tweet id) → normalized list of every tweet in the thread (focal tweet + replies + sub-replies), with author, engagement, media, cursor.

Language

All process output to user (progress updates, process notifications) follows the user's language.

Objective

Given an X conversation id (which equals the root tweet id), collect the focal tweet and every reply / sub-reply visible to the logged-in session, returning structured per-tweet data with pagination cursors.

Prerequisites

• Active X session in the browser (left sidebar shows logged-in avatar / @handle).
• Network capture is enabled in the browser-act session.

Pre-execution Checks

1. Tool Readiness

If browser-act has been confirmed available in the current session → skip this step.

Invoke browser-act via Skill tool to load usage. If installation or configuration issues arise, follow its guidance to resolve then retry.

2. Login Verification

If login status for X has been confirmed in the current session → skip this step.

Otherwise: open https://x.com and observe the left sidebar:

• User avatar or @handle visible → logged in, continue
• "Sign in" / "Log in" prompt visible → not logged in, inform the user and assist the login flow

User refuses or cannot log in → terminate execution.

Capability Components

This Skill's operational boundary = what the user can manually do in their browser. It only reads tweet data already shown to the user, never bypassing authentication. The browser's own JS signs the GraphQL request; the Skill triggers it via URL navigation and reads the response from network traffic. Python scripts under scripts/ only build URLs and parse responses — they do not call X directly. Run them through the bash tool.

Network Capture: full conversation thread

Step 1 — build the tweet-detail URL for the conversation root:

URL=$(python scripts/build-conversation-url.py '{conversation_id}' [--handle {handle}])

Parameters:

• conversation_id (positional): the X conversation id, which equals the root tweet id (the same number you would see in https://x.com/i/status/<conversation_id>).
• --handle: the root tweet's author handle if known; if unknown, leave the default — x.com/i/status/<id> resolves to the same focal tweet because X canonicalises the URL.

Step 2 — navigate and capture the first page:

1. network requests --clear
2. navigate "$URL"
3. wait stable --timeout 25000 (timeout is normal on X; proceed)
4. network requests --type xhr,fetch --filter TweetDetail → take the latest entry's request_id
5. network request <request_id> → save full output to a file (e.g. tmp/x-conversation-page-1.txt)
6. python scripts/parse-tweets.py --json-file tmp/x-conversation-page-1.txt --source tweet_detail → emits JSON {tweets, count, cursor_top, cursor_bottom}. The focal tweet is the first element; subsequent elements are replies (in display order). Each reply carries is_reply: true, in_reply_to_id, in_reply_to_user, and conversation_id == <root tweet id>, which makes the reply tree reconstructable downstream.

Endpoint characteristic: URL contains /i/api/graphql/<hash>/TweetDetail. The query hash rotates; always filter by name.

Step 3 — paginate via scroll to load deeper replies and sub-replies:

1. network requests --clear
2. scroll down --amount 5000
3. wait stable --timeout 10000
4. network requests --type xhr,fetch --filter TweetDetail → newest entry's request_id
5. network request <request_id> → save to tmp/x-conversation-page-N.txt
6. python scripts/parse-tweets.py --json-file tmp/x-conversation-page-N.txt --source tweet_detail

Repeat Step 3 until any termination condition is met:

• Accumulated unique reply count reaches the user's target.
• count == 0 on the current page.
• cursor_bottom is unchanged across two consecutive pages.
• The page shows a "Show more replies" button gated by visibility filters — state reveals it; the Agent may click <index> to expand more, then resume scrolling.

Error handling:

• If the focal tweet is deleted or protected, the response carries a TweetTombstone entry; parse-tweets.py filters tombstones, so the result will be count: 0 — report to the user and stop.
• If no TweetDetail request appears after a scroll, wait 3 s and retry once. Persistent absence after two scrolls means the thread has been fully loaded.
• If a captcha challenge appears (visible in state as an authorization prompt), pause and ask the user.

Output example:

{
  "tweets": [
    {
      "type": "tweet",
      "id": "2069990565530214798",
      "url": "https://x.com/Rothmus/status/2069990565530214798",
      "twitter_url": "https://twitter.com/Rothmus/status/2069990565530214798",
      "text": "\"We are going to have a multi-racial nation in Singapore. ...\"",
      "created_at": "Tue Jun 24 23:01:50 +0000 2026",
      "lang": "en",
      "source": "Twitter Web App",
      "retweet_count": 91,
      "reply_count": 84,
      "like_count": 567,
      "quote_count": 12,
      "bookmark_count": 50,
      "view_count": 64140,
      "is_reply": false,
      "is_retweet": false,
      "is_quote": false,
      "quote_id": null,
      "quote_url": null,
      "in_reply_to_id": null,
      "in_reply_to_user": null,
      "in_reply_to_user_id": null,
      "conversation_id": "2069990565530214798",
      "hashtags": [],
      "mentions": [],
      "urls": [],
      "media": [],
      "card": null,
      "place": null,
      "author": {
        "id": "987654321",
        "user_name": "Rothmus",
        "name": "Rothmus",
        "url": "https://x.com/Rothmus",
        "is_verified": false,
        "is_blue_verified": true,
        "verified_type": null,
        "profile_picture": "https://pbs.twimg.com/profile_images/.../photo.jpg",
        "description": "Singapore observer ...",
        "location": "Singapore",
        "followers": 12345,
        "following": 678,
        "created_at": "Wed Jun 16 10:00:00 +0000 2021"
      }
    },
    {
      "type": "tweet",
      "id": "2070027327409655973",
      "url": "https://x.com/DinoLeadingNews/status/2070027327409655973",
      "text": "@Rothmus @elonmusk Singapore proves that identity ...",
      "is_reply": true,
      "in_reply_to_id": "2069990565530214798",
      "in_reply_to_user": "Rothmus",
      "in_reply_to_user_id": "987654321",
      "conversation_id": "2069990565530214798",
      "like_count": 0,
      "reply_count": 0,
      "retweet_count": 0,
      "quote_count": 0,
      "bookmark_count": 0,
      "view_count": 19,
      "author": {"user_name": "DinoLeadingNews", "name": "Dino Leading News", "...": "..."},
      "...": "..."
    }
  ],
  "count": 40,
  "cursor_top": null,
  "cursor_bottom": "DAACCgABHLoz..."
}

Pagination

Network Capture Pagination: triggered by scroll down. X's page JS inserts the previous response's cursor_bottom into the next TweetDetail request's variables.cursor. Some sub-reply branches require clicking an in-page "Show replies" expander (state to locate, click <index>) before the next scroll surfaces them. Termination: count == 0, cursor_bottom does not advance across two consecutive pages, or user target reached.

Success Criteria

count >= 1 on the first page (the focal tweet must be present unless the conversation root is deleted) AND every tweet has non-null id, text, created_at, author.user_name, like_count, retweet_count, reply_count, conversation_id AND each reply tweet has is_reply == true and in_reply_to_id pointing to a tweet inside the thread.

Known Limitations

• Only replies visible to the logged-in session are returned; X enforces visibility filters (blocked accounts, soft-blocked replies, "Show additional replies, including those that may contain offensive content").
• Deleted root tweets return a tombstone; the Skill terminates with count: 0.
• Quote tweets that reference this conversation are NOT included — they live in a separate timeline; collect them via x-tweet-search-by-query with filter:quote conversation_id:<id> if needed.
• view_count is null for very new replies where X has not emitted views.count.
• Sustained polling triggers per-session throttling; stay under ~150 timeline calls per 15-minute window per session.
• The page may never reach network-idle; wait stable will frequently time out — proceed to read network anyway.

Execution Efficiency

• Batch orchestration: write a bash script that iterates conversation ids serially in one session. For parallelism, fan out across multiple stealth browsers each with its own login.
• Test before batch execution: run one conversation end-to-end (page 1 + at least one paginated page) first.
• Reduce redundant pre-operations: keep the same session for many sequential conversations.
• Error resumption: persist cursor_bottom and accumulated reply IDs per conversation after every page.
• De-duplicate by id: the focal tweet may appear in multiple paginated responses (some endpoints repeat the focal tweet at the top of every page); merge by id.

Experience Notes

Path: {working-directory}/browser-act-skill-forge-memories/x-tweet-scraper-x-tweet-by-conversation.memory.md (working directory is determined by the Agent running the Skill, typically the project root or current working directory)

Before execution: If the file exists, read it first — it records unexpected situations encountered during past executions (e.g., a strategy has become ineffective); adjust strategy order accordingly.

After execution: If an unexpected situation is encountered (strategy became ineffective, page redesigned, anti-scraping upgraded, better path discovered), append a line: {YYYY-MM-DD}: {what happened} → {conclusion}

Normal execution does not write to the file. Do not record which conversation was scraped or how many replies were returned — those are task outputs, not experience.