social-scraping

# Social Media Scraping with StableSocial Scrape profiles, posts, comments, followers, and search across 20+ platforms: TikTok, Instagram, YouTube, LinkedIn, X/Twitter, Facebook, Reddit, GitHub, Rumble, Threads, Bluesky, Pinterest, Twitch, Spotify, and more — plus ad libraries and a UGC research agent (Lightreel). All endpoints cost $0.06 per call. Three endpoint families: - **Legacy platform endpoints** (`/api/tiktok/*`, `/api/instagram/*`, `/api/facebook/*`, `/api/reddit/*`) — profiles, posts, comments, followers, search - **Scrape Creators suite** (`/api/sc/*`) — much broader platform and data coverage, including transcripts, ad libraries, and niche platforms - **Lightreel UGC research agent** (`/api/lightreel/*`) — durable-job research tasks (hooks, trends, creator search, scripts, briefs) ## Setup See [rules/getting-started.md](rules/getting-started.md) for installation and wallet setup. ## Notes Use `npx agentcash@latest fetch` for paid POST triggers. Use `npx agentcash@latest fetch` for free GET polling. IMPORTANT: Use exact endpoint paths from the Quick Reference tables below. All paths include a platform prefix (e.g. `https://stablesocial.dev/api/tiktok/...`). ## How It Works: Async Two-Step Flow Every request follows a **trigger-then-poll** pattern: ### Step 1: Trigger (paid, $0.06) ```bash npx agentcash@latest fetch https://stablesocial.dev/api/instagram/profile -m POST -b '{"handle": "natgeo"}' ``` Returns `202 Accepted` with a durable job ID, poll URL, and legacy JWT token: ```json {"jobId": "abc123", "status": "pending", "pollUrl": "https://stablesocial.dev/api/jobs/abc123", "token": "eyJhbGciOiJIUzI1NiIs..."} ``` ### Step 2: Poll (free) Poll the durable job by ID (requires SIWX wallet auth from the paying wallet — the agentcash CLI handles this automatically): ```bash npx agentcash@latest fetch https://stablesocial.dev/api/jobs/abc123 ``` Or poll with the legacy token (no wallet auth needed): ```bash npx agentcash@latest fetch "https://stablesocial.dev/api/jobs?token=eyJhbGciOiJIUzI1NiIs..." ``` - `{"status": "pending"}` — poll again in 3-5 seconds - `{"status": "finished", "data": {...}}` — data is ready - `{"status": "failed", "error": "..."}` — collection failed (not charged) Tokens expire after 30 minutes; durable jobs remain pollable by jobId. Jobs typically finish in 5-60 seconds (Lightreel jobs can take longer). `GET /api/jobs` (no token) lists your durable jobs. ## Quick Reference — TikTok | Task | Endpoint | Depends On | |------|----------|------------| | Get profile | `https://stablesocial.dev/api/tiktok/profile` | — | | Get posts | `https://stablesocial.dev/api/tiktok/posts` | profile | | Post comments | `https://stablesocial.dev/api/tiktok/post-comments` | posts | | Comment replies | `https://stablesocial.dev/api/tiktok/comment-replies` | post-comments | | Followers | `https://stablesocial.dev/api/tiktok/followers` | profile | | Following | `https://stablesocial.dev/api/tiktok/following` | profile | | Search posts | `https://stablesocial.dev/api/tiktok/search` | — | | Search hashtag | `https://stablesocial.dev/api/tiktok/search-hashtag` | — | | Search profiles | `https://stablesocial.dev/api/tiktok/search-profiles` | — | | Search by music | `https://stablesocial.dev/api/tiktok/search-music` | — | **Input:** `{"handle": "username"}` for profile/posts/followers. `{"query": "keyword"}` for search. ## Quick Reference — Instagram | Task | Endpoint | Depends On | |------|----------|------------| | Get profile | `https://stablesocial.dev/api/instagram/profile` | — | | Get posts | `https://stablesocial.dev/api/instagram/posts` | profile | | Post comments | `https://stablesocial.dev/api/instagram/post-comments` | posts | | Comment replies | `https://stablesocial.dev/api/instagram/comment-replies` | post-comments | | Followers | `https://stablesocial.dev/api/instagram/followers` | profile | | Following | `https://stablesocial.dev/api/instagram/following` | profile | | Stories | `https://stablesocial.dev/api/instagram/stories` | profile | | Highlights | `https://stablesocial.dev/api/instagram/highlights` | profile | | Search posts | `https://stablesocial.dev/api/instagram/search` | — | | Search tags | `https://stablesocial.dev/api/instagram/search-tags` | — | **Input:** `{"handle": "username"}` for profile/posts/followers. `{"query": "keyword"}` for search. ## Quick Reference — Facebook | Task | Endpoint | Depends On | |------|----------|------------| | Get profile | `https://stablesocial.dev/api/facebook/profile` | — | | Get posts | `https://stablesocial.dev/api/facebook/posts` | profile | | Post comments | `https://stablesocial.dev/api/facebook/post-comments` | posts | | Comment replies | `https://stablesocial.dev/api/facebook/comment-replies` | post-comments | | Followers | `https://stablesocial.dev/api/facebook/followers` | profile | | Following | `https://stablesocial.dev/api/facebook/following` | profile | | Search posts | `https://stablesocial.dev/api/facebook/search` | — | | Search people | `https://stablesocial.dev/api/facebook/search-people` | — | | Search pages | `https://stablesocial.dev/api/facebook/search-pages` | — | | Search groups | `https://stablesocial.dev/api/facebook/search-groups` | — | **Input:** `{"handle": "username"}` or `{"profile_id": "id"}` for profile. `{"query": "keyword"}` for search. ## Quick Reference — Reddit | Task | Endpoint | Depends On | |------|----------|------------| | Get post | `https://stablesocial.dev/api/reddit/post` | — | | Post comments | `https://stablesocial.dev/api/reddit/post-comments` | post | | Get comment | `https://stablesocial.dev/api/reddit/comment` | — | | Search posts | `https://stablesocial.dev/api/reddit/search` | — | | Search profiles | `https://stablesocial.dev/api/reddit/search-profiles` | — | | Subreddit posts | `https://stablesocial.dev/api/reddit/subreddit` | — | **Input:** `{"post_id": "id"}` for post details. `{"query": "keyword"}` for search. `{"subreddit": "name"}` for subreddit. ## Quick Reference — Scrape Creators Suite (`/api/sc/*`) A larger, newer suite (~155 endpoints, all $0.06, same trigger-then-poll flow) covering many more platforms and data types. Coverage by platform: | Platform | Base path | Coverage highlights | |----------|-----------|---------------------| | TikTok | `https://stablesocial.dev/api/sc/tiktok/...` | profile, videos, video transcripts, comments/replies, followers/following, audience demographics, search (users/hashtag/keyword/top), trending feed, popular creators/hashtags, songs, Shop products/reviews | | Instagram | `https://stablesocial.dev/api/sc/instagram/...` | profile, posts, reels, post/reel info, media transcripts, comments, highlights, hashtag/profile/reels search, trending reels | | YouTube | `https://stablesocial.dev/api/sc/youtube/...` | channel details/videos/playlists/lives/shorts/community posts, video details, transcripts, sponsors, comments/replies, search, trending shorts | | LinkedIn | `https://stablesocial.dev/api/sc/linkedin/...` | person profile (pass a linkedin.com/in/... URL), company page/posts, post details/transcript, search posts, ads search/details | | Facebook | `https://stablesocial.dev/api/sc/facebook/...` | profile, posts/reels/photos, post transcript, comments/replies, group posts, Ad Library (search/details/transcript/company ads), Marketplace (search/item/location), Events (search/details) | | X/Twitter | `https://stablesocial.dev/api/sc/twitter/...` | profile, user tweets, tweet details/transcript, communities | | Reddit | `https://stablesocial.dev/api/sc/reddit/...` | subreddit details/posts/search, post comments/transcript, search | | GitHub | `https://stablesocial.dev/api/sc/github/...` | user, repositories, PRs, activity, followers/following, contributions, repository, trending repos/developers | | Rumble | `https://stablesocial.dev/api/sc/rumble/...` | search, channel videos, video, transcript, comments | | Ad libraries | `.../api/sc/tiktok/ad-library/...`, `.../api/sc/google/...`, `.../api/sc/linkedin/ads/...` | TikTok Ad Library search/ad, Google company ads/ad details/advertiser search, LinkedIn ads | | Others | `https://stablesocial.dev/api/sc/...` | Truth Social, Threads, Bluesky, Pinterest, Twitch, Spotify, SoundCloud, Kwai, Kick, Snapchat, Google Search, Amazon Shop, link-in-bio pages (Linktree, Komi, Pillar, Linkbio, Linkme), age/gender detection | Representative examples: ```bash npx agentcash@latest fetch https://stablesocial.dev/api/sc/tiktok/profile -m POST -b '{"handle": "username"}' npx agentcash@latest fetch https://stablesocial.dev/api/sc/youtube/video/transcript -m POST -b '{"url": "https://youtube.com/watch?v=..."}' npx agentcash@latest fetch https://stablesocial.dev/api/sc/linkedin/profile -m POST -b '{"url": "https://linkedin.com/in/someone"}' npx agentcash@latest fetch https://stablesocial.dev/api/sc/facebook/adLibrary/search/ads -m POST -b '{"query": "brand name"}' ``` **Input:** varies per endpoint — many accept `{"handle": ...}` or an ID, some accept a full `{"url": ...}` (e.g. LinkedIn profile). Check the endpoint schema (`npx agentcash@latest discover https://stablesocial.dev`) when unsure. ## Quick Reference — Lightreel UGC Research Agent (`/api/lightreel/*`) An AI research agent for UGC/short-form content. All $0.06 per call; returns a durable job immediately — poll `GET /api/jobs/{jobId}`. Jobs are research tasks and can take minutes, not seconds. | Task | Endpoint | |------|----------| | Ask the agent anything | `https://stablesocial.dev/api/lightreel/chat` | | Top-performing hooks | `https://stablesocial.dev/api/lightreel/top-hooks` | | UGC video search | `https://stablesocial.dev/api/lightreel/video-search` | | Creator search (incl. contact info) | `https://stablesocial.dev/api/lightreel/creator-search` | | Content trends | `https://stablesocial.dev/api/lightreel/trends` | | Account audit/feedback | `https://stablesocial.dev/api/lightreel/account-feedback` | | Competitor strategy | `https://stablesocial.dev/api/lightreel/competitor-strategy` | | Script ideas | `https://stablesocial.dev/api/lightreel/script-ideas` | | Video ideas | `https://stablesocial.dev/api/lightreel/video-ideas` |