agent-reach-internet-access

name	agent-reach-internet-access
description	Give AI agents eyes to see the internet — scrape Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu with zero API fees
triggers	["search twitter for mentions","get youtube video transcript","scrape this reddit thread","read this xiaohongshu post","search github repositories","get bilibili video subtitles","read this web page content","search the internet for"]

Agent Reach — Internet Access for AI Agents

Skill by ara.so — AI Agent Skills collection.

Agent Reach is a scaffolding tool that gives AI agents the ability to read and search across Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, and more — all without paid APIs. It orchestrates best-in-class upstream tools (yt-dlp, twitter-cli, rdt-cli, gh CLI, etc.) and provides a unified interface for AI agents.

Installation

Agent Reach is installed via pip and automatically sets up dependencies:

# Basic installation
pip install agent-reach

# The tool will auto-detect and install:
# - Node.js (for some MCP servers)
# - gh CLI (for GitHub)
# - mcporter (for MCP integrations)
# - twitter-cli (for Twitter/X)
# - rdt-cli (for Reddit)
# - yt-dlp (for YouTube/Bilibili)

After installation, run diagnostics to check what's working:

agent-reach doctor

This shows status for each channel: ✅ (works out of box), 🔧 (needs config), or ❌ (not available).

Core Capabilities

1. Web Page Reading (Jina Reader)

Read any web page as clean markdown:

# Read a web page
curl https://r.jina.ai/https://example.com

# Get JSON format
curl https://r.jina.ai/https://example.com \
  -H "Accept: application/json"

# With images
curl https://r.jina.ai/https://example.com \
  -H "X-With-Images-Summary: true"

Python usage:

import requests

url = "https://example.com"
response = requests.get(f"https://r.jina.ai/{url}")
markdown_content = response.text

# With options
headers = {
    "X-With-Links-Summary": "true",
    "X-With-Images-Summary": "true"
}
response = requests.get(f"https://r.jina.ai/{url}", headers=headers)

2. YouTube & Video (yt-dlp)

Extract subtitles, metadata, and search videos:

# Get video metadata + subtitles
yt-dlp --dump-json --write-auto-subs --skip-download \
  "https://www.youtube.com/watch?v=VIDEO_ID"

# Search YouTube
yt-dlp "ytsearch5:AI agents tutorial" --dump-json

# Get specific subtitle language
yt-dlp --write-subs --sub-lang en --skip-download URL

# Bilibili videos (works same way)
yt-dlp --dump-json "https://www.bilibili.com/video/BV..."

Python usage:

import subprocess
import json

def get_video_info(url):
    result = subprocess.run(
        ["yt-dlp", "--dump-json", "--write-auto-subs", 
         "--skip-download", url],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)

# Search videos
def search_youtube(query, max_results=5):
    result = subprocess.run(
        ["yt-dlp", f"ytsearch{max_results}:{query}", "--dump-json"],
        capture_output=True, text=True
    )
    return [json.loads(line) for line in result.stdout.strip().split('\n')]

3. Twitter/X (twitter-cli)

Requires Cookie authentication. Export cookies using Cookie-Editor Chrome extension.

# Configure (paste exported cookies when prompted)
twitter configure

# Read a tweet
twitter tweet https://twitter.com/user/status/123456789

# Search tweets
twitter search "AI agents" --limit 20

# Get user timeline
twitter timeline @username --limit 50

# Get tweet thread
twitter thread https://twitter.com/user/status/123456789

Configuration file location: ~/.twitter-cli/config.json

4. Reddit (rdt-cli)

Requires Cookie authentication:

# Login with cookies
rdt login

# Search posts
rdt search "machine learning" --limit 20

# Read post with comments
rdt post https://reddit.com/r/programming/comments/...

# Get subreddit posts
rdt subreddit r/python --limit 30

5. GitHub (gh CLI)

# Login (opens browser OAuth flow)
gh auth login

# View repository
gh repo view owner/repo

# Search repositories
gh search repos "LLM framework" --limit 20

# Search issues
gh search issues "bug" --repo owner/repo

# View issue
gh issue view 123 --repo owner/repo

# Create issue
gh issue create --repo owner/repo \
  --title "Bug report" --body "Description"

Python usage:

import subprocess
import json

def search_repos(query, limit=20):
    result = subprocess.run(
        ["gh", "search", "repos", query, 
         "--limit", str(limit), "--json", "name,description,url"],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)

def get_repo_info(owner_repo):
    result = subprocess.run(
        ["gh", "repo", "view", owner_repo, "--json", 
         "description,stargazerCount,forkCount,url"],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)

6. XiaoHongShu (xhs-cli via mcporter)

Requires Cookie authentication:

# Configure (sets up MCP server)
mcporter add xiaohongshu

# The MCP server provides these tools:
# - search_notes: Search XHS posts
# - get_note_detail: Get post content
# - post_note: Create new post
# - comment_note: Add comment
# - like_note: Like a post

Configuration stored in: ~/.mcporter/xiaohongshu/config.json

7. Bilibili Enhanced (bili-cli)

# Get hot videos
bili hot --limit 20

# Search videos
bili search "Python tutorial" --limit 30

# Get video info
bili video BV1xx411c7mD

# Get user dynamics
bili user-dynamic 123456

8. Internet Search (Exa via mcporter)

Semantic search across the web:

# Add Exa MCP server (no API key needed for basic use)
mcporter add exa

# The MCP server provides:
# - search: AI-powered semantic search
# - find_similar: Find similar pages
# - get_contents: Extract page contents

For advanced features, set API key:

export EXA_API_KEY=your_key_here

9. RSS Feeds

import feedparser

# Parse RSS feed
feed = feedparser.parse("https://example.com/feed.xml")

for entry in feed.entries:
    print(f"Title: {entry.title}")
    print(f"Link: {entry.link}")
    print(f"Published: {entry.published}")
    print(f"Summary: {entry.summary}")
    print("---")

10. WeChat Official Accounts

Search and read WeChat articles via Exa + Camoufox:

# Use Exa search to find WeChat articles
# Articles are auto-extracted when URLs contain mp.weixin.qq.com

11. Weibo (微博)

# Search content
agent-reach weibo search "AI" --type content

# Get hot search
agent-reach weibo hot

# Get user posts
agent-reach weibo user USER_ID

# Get comments
agent-reach weibo comments POST_ID

12. V2EX

# Get hot topics
agent-reach v2ex hot

# Get node topics
agent-reach v2ex node python

# Get topic details
agent-reach v2ex topic 123456

Configuration Patterns

Cookie-Based Services

For Twitter, Reddit, XiaoHongShu — use Cookie-Editor:

1. Login to the service in browser
2. Install Cookie-Editor
3. Click extension → Export → Copy
4. Paste into CLI config command

Never commit cookies to version control. They're stored in:

• Twitter: ~/.twitter-cli/config.json
• Reddit: ~/.rdt-cli/cookies.json
• XHS: ~/.mcporter/xiaohongshu/config.json

Proxy Configuration (Server Deployments)

For Bilibili access from servers:

# Set proxy environment variables
export HTTP_PROXY=http://proxy-server:port
export HTTPS_PROXY=http://proxy-server:port

# Or configure per-tool
yt-dlp --proxy http://proxy-server:port URL

GitHub Authentication

# OAuth login (recommended)
gh auth login

# Or use token
export GITHUB_TOKEN=ghp_your_token_here
gh auth login --with-token <<< $GITHUB_TOKEN

Common Workflows

Scrape Twitter Thread for Research

import subprocess
import json

def get_twitter_thread(url):
    result = subprocess.run(
        ["twitter", "thread", url],
        capture_output=True, text=True
    )
    return result.stdout

thread_content = get_twitter_thread(
    "https://twitter.com/user/status/123456789"
)

Extract YouTube Video Summary

import subprocess
import json

def get_video_transcript(url):
    # Get metadata + subtitles
    result = subprocess.run(
        ["yt-dlp", "--dump-json", "--write-auto-subs", 
         "--skip-download", url],
        capture_output=True, text=True
    )
    data = json.loads(result.stdout)
    
    # Subtitles are in data['subtitles'] or data['automatic_captions']
    return {
        'title': data.get('title'),
        'description': data.get('description'),
        'duration': data.get('duration'),
        'subtitles': data.get('automatic_captions', {})
    }

Search GitHub for Solutions

import subprocess
import json

def search_github_issues(query, repo=None):
    cmd = ["gh", "search", "issues", query, 
           "--limit", "20", "--json", 
           "title,url,state,body,comments"]
    if repo:
        cmd.extend(["--repo", repo])
    
    result = subprocess.run(cmd, capture_output=True, text=True)
    return json.loads(result.stdout)

# Search across all repos
issues = search_github_issues("memory leak in agents")

# Search specific repo
issues = search_github_issues("bug", repo="openai/gpt-4")

Monitor Reddit for Mentions

# Search and save results
rdt search "your_product_name" --limit 50 > mentions.txt

# Get specific subreddit
rdt subreddit r/artificial --limit 100

Read Web Page Content for Analysis

import requests

def get_clean_content(url):
    response = requests.get(
        f"https://r.jina.ai/{url}",
        headers={
            "X-With-Links-Summary": "true",
            "X-No-Cache": "true"
        }
    )
    return response.text

content = get_clean_content("https://news.ycombinator.com")

Troubleshooting

Doctor Command Shows ❌

Run diagnostics:

agent-reach doctor

Each ❌ includes a fix suggestion. Common issues:

Twitter/Reddit not working:

• Need Cookie authentication
• Use Cookie-Editor to export cookies
• Run twitter configure or rdt login

Bilibili 403 on server:

• Need proxy for non-CN IPs
• Set HTTP_PROXY and HTTPS_PROXY env vars

GitHub rate limited:

• Authenticate: gh auth login
• Authenticated rate: 5,000/hour vs 60/hour

yt-dlp fails:

• Update to latest: pip install -U yt-dlp
• Tool is actively maintained, updates frequently

MCP Server Connection Issues

# Check mcporter status
mcporter list

# Restart a server
mcporter restart xiaohongshu

# View server logs
mcporter logs exa

Proxy Not Working

# Test proxy connection
curl -x http://proxy:port https://api.bilibili.com

# Set for specific command
export HTTPS_PROXY=http://proxy:port
yt-dlp URL

Cookie Expired

Re-export fresh cookies:

1. Login to service in browser
2. Export with Cookie-Editor
3. Reconfigure CLI tool

Environment Variables

# Proxy (for server deployments)
export HTTP_PROXY=http://proxy:port
export HTTPS_PROXY=http://proxy:port

# GitHub
export GITHUB_TOKEN=ghp_xxxxx

# Exa (optional, for advanced features)
export EXA_API_KEY=your_key_here

# Custom config paths (optional)
export AGENT_REACH_CONFIG_DIR=~/.config/agent-reach

Safety & Privacy

• All cookies stored locally in ~/.twitter-cli/, ~/.rdt-cli/, etc.
• No data uploaded to Agent Reach servers (there are none)
• Code is open source — audit anytime
• Use --safe mode during install to review system package installs

Updating

# Update agent-reach
pip install -U agent-reach

# Update individual tools
pip install -U yt-dlp
gh extension upgrade --all
npm update -g mcporter

Check for breaking changes: https://github.com/Panniantong/agent-reach/blob/main/CHANGELOG.md

Platform Support Matrix

Platform	Out of Box	After Config	Notes
Web	✅	—	Jina Reader, no limits
YouTube	✅	—	yt-dlp, 1800+ sites
RSS	✅	—	feedparser
GitHub	✅	🔧 Auth for private	gh CLI
Twitter	🔧 Cookie	🔧 Cookie	twitter-cli
Reddit	🔧 Cookie	🔧 Cookie	rdt-cli
Bilibili	✅ Local	🔧 Proxy (server)	yt-dlp
XiaoHongShu	🔧 Cookie	🔧 Cookie	xhs-cli via MCP
Search	🔧 MCP	🔧 API key (optional)	Exa
WeChat	✅	—	Via Exa search
Weibo	✅	—	Direct API
V2EX	✅	—	Direct API

Legend: ✅ Works immediately | 🔧 Needs configuration