read-forums

Use when building scrapers to extract and index forum discussions, blog posts, or community content into markdown files with postgres/vector search integration. Triggers on requests to scrape, sync, index, or extract forum content.

Zur Installation springen

Repository: StrongAI/claude-skills-reading-forums
Letzte Quellaktivität: 17. März 2026 um 00:20
Erkannte Sprache von SKILL.md: Englisch
Sterne: 0
Forks: 0

Installationsoptionen

Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.

Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.

Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.

npx skills add https://github.com/StrongAI/claude-skills-reading-forums --skill read-forums

Der Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.

Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.

Quelldateien prüfen

Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.

SKILL.md

Quellanweisungen · Schreibgeschützte Vorschau

Zurück zur Suche Zurück zur Installation

name	read-forums
description	Use when building scrapers to extract and index forum discussions, blog posts, or community content into markdown files with postgres/vector search integration. Triggers on requests to scrape, sync, index, or extract forum content.

Forum Extraction & Indexing

Overview

Build scrapers that extract forum discussions into structured markdown files, then index them for semantic search. Each forum platform (Discourse, phpBB, Drupal, Vanilla, static blogs) needs a platform-specific fetcher, but the output format, rate limiting, incremental sync, and indexing pipeline are universal.

Core principle: Be a good citizen. Rate-limit aggressively, respect robots.txt, use APIs when available, and never hit a forum harder than a human would browse it.

Architecture

Forum API/HTML ──→ Platform Fetcher ──→ Markdown + YAML Frontmatter ──→ File Storage
                                                                            │
                                                                    Statement-MCP / QMD
                                                                     (semantic search)

Every scraper follows the same pipeline:

Fetch — Platform-specific API calls or HTML scraping
Transform — Convert to markdown with structured YAML frontmatter
Store — One file per discussion/post, organized by category
Track — Incremental sync state (last-fetched timestamps)
Index — Ingest into statement-mcp or QMD for semantic search

Output Format

Every discussion/post becomes a markdown file with YAML frontmatter:

---
id: 12345
title: "Discussion title"
category: category-slug
author: username
created: '2025-01-15T10:30:00+00:00'
updated: '2025-01-16T08:00:00+00:00'
tags: [nurbs, surface, continuity]
url: https://forum.example.com/t/12345
source: forum-name
platform: discourse
countReplies: 14
countViews: 892
solved: true
---

# Discussion title

[body as markdown]

---

## Replies

### username — 2025-01-15T11:00:00+00:00

[reply body as markdown]

Required frontmatter fields: id, title, category, author, created, url, source, platform Optional but recommended: tags, countReplies, countViews, solved/accepted, updated

File Organization

{output-dir}/
├── categories/
│   ├── {category-slug}/
│   │   ├── {id}.md
│   │   └── ...
│   └── ...
├── metadata/
│   ├── sync-state.json    ← last sync timestamp per category
│   └── categories.json    ← category list cache
└── README.md              ← auto-generated stats

Rate Limiting

Non-negotiable rules:

Forum size	Delay between requests	Why
Small (<1K)	3-5 seconds	Don't overwhelm small communities
Medium (1-10K)	2-3 seconds	Sustainable pace
Large (10K+)	1-2 seconds	API can handle it; still be respectful
Blog (static)	1-2 seconds	Low load, but don't slam

Always:

Use a RateLimiter class (see reference implementation below)
Set a User-Agent identifying the scraper: OnshapeAssistant/1.0 (research; contact@example.com)
Respect Retry-After headers on 429 responses
Back off exponentially on errors (start 5s, max 60s)
Stop entirely on 403 or repeated 429s

Platform-Specific Fetchers

Discourse (McNeel Forum, Blender devtalk)

API: Discourse has a public JSON API. Append .json to any URL.

GET /categories.json                              → list categories
GET /c/{slug}/{id}.json?page=N                    → topics in category
GET /t/{topic_id}.json                            → topic with posts
GET /t/{topic_id}/posts.json?post_ids[]=N&...     → specific posts

Pagination: Topics list paginated by page param. Posts within a topic may need chunked fetching for long threads (post stream IDs in topic JSON).

Key fields: topic.title, topic.posts_count, topic.views, topic.tags, topic.created_at, post.cooked (HTML body), post.username, post.created_at

Rate limit: Discourse default is 200 req/min for anonymous. Use 2s delay to stay well under.

phpBB (FreeCAD Forum)

No API. Must scrape HTML.

GET /viewforum.php?f={forum_id}&start={offset}    → topic list (25/page)
GET /viewtopic.php?t={topic_id}&start={offset}    → posts in topic (15/page)

HTML selectors:

Topic list: .topiclist .row or similar (varies by theme)
Post body: .postbody .content
Post author: .postprofile .username
Pagination: .pagination links

Key gotcha: phpBB HTML varies significantly by theme. Test selectors against the specific forum before bulk scraping.

Drupal (Open CASCADE Dev Forum)

No standard API. Scrape HTML.

GET /forums/{forum_id}?page={N}                   → topic list
GET /node/{node_id}                                → single topic with comments

HTML selectors (typical Drupal):

Topic list: .view-content .views-row
Topic body: .field--name-body
Comments: .comment-wrapper .comment

Key gotcha: Older Drupal sites may have different markup. Some content may be at old.opencascade.com with a different structure.

Static Blogs (Quaoar, C3D Labs, Geometric Tools)

Approach: Sitemap or manual URL list → fetch each page → extract article content.

GET /sitemap.xml                                   → all URLs
GET /feed/ or /rss/                                → RSS feed with recent posts
GET /{post-url}                                    → individual article

HTML selectors: Highly variable. Use <article> tag or main content area. Strip nav, sidebar, footer.

For Quaoar specifically: Small volume (~100 posts), extraordinary depth. Can be fully indexed in one pass with no incremental sync needed.

Incremental Sync

Track last-sync timestamps per category in metadata/sync-state.json:

{
  "developer": "2026-03-08T10:00:00+00:00",
  "grasshopper": "2026-03-08T10:15:00+00:00"
}

Sync logic:

Load sync state
For each category, fetch discussions updated since last sync
Overwrite existing files (they may have new replies)
Update sync state after each category completes
Save state atomically (write to .tmp, rename)

For APIs (Discourse, Vanilla): Use sort-by-latest and stop paginating when you hit posts older than the cutoff.

For HTML scrapers (phpBB, Drupal): Fetch first N pages of each forum (sorted by last post date), stop when you see topics older than cutoff.

Indexing into Statement-MCP

After scraping, ingest the collection:

# Using statement-mcp's ingest_collection
mcp__statement-mcp__ingest_collection(
    path="/path/to/forum/categories",
    collection="mcneel-forum",
    glob_pattern="**/*.md"
)

Re-running is safe: ingest_collection uses content hashes to skip unchanged files and marks deleted files as superseded.

CLI Pattern

Use Click for CLI commands. Each forum gets its own command:

@cli.command()
@click.option("--output-dir", required=True, type=click.Path())
@click.option("--delay", default=2.0, help="Seconds between API requests")
@click.option("--since", default=None, help="ISO timestamp cutoff")
@click.option("--category", default=None, help="Sync only this category")
@click.option("--skip-replies", is_flag=True, help="Skip fetching replies")
@click.option("--max-pages", default=None, type=int, help="Max pages per category")
def sync_mcneel(output_dir, delay, since, category, skip_replies, max_pages):
    """Sync McNeel/Grasshopper forum discussions."""
    ...

Always include --max-pages for initial testing and to cap runtime.

Reference Implementation

The Onshape forum syncer at mcp/tools/tools/sync_forums.py is the reference:

273 lines, Vanilla Forums API v2
RateLimiter class in utils.py
html_to_markdown() using markdownify + BeautifulSoup
YAML frontmatter with engagement metadata
Incremental sync via sync-state.json
Click CLI with --delay, --since, --category, --skip-comments

Adapt this pattern for each new platform. The output format and sync infrastructure are reusable; only the API/HTML fetching layer changes.

Checklist for Adding a New Forum

Common Mistakes

Mistake	Fix
No rate limiting	Always use RateLimiter. 2s minimum.
Scraping without checking robots.txt	Check first. Respect disallow rules.
Storing raw HTML	Convert to markdown. HTML is noisy and wastes tokens.
No incremental sync	Always track timestamps. Full re-scrape is wasteful.
Missing frontmatter	Every file needs at minimum: id, title, author, created, url, source
Giant monolithic scraper	One command per platform. Share utils.
No error handling on network	Retry with exponential backoff. Log failures. Continue.
Running at full speed on small forums	3-5s delay for <1K post forums

read-forums

Quellinformationen

Installationsoptionen

Quelldateien prüfen

Forum Extraction & Indexing

Overview

Architecture

Output Format

File Organization

Rate Limiting

Platform-Specific Fetchers

Discourse (McNeel Forum, Blender devtalk)

phpBB (FreeCAD Forum)

Drupal (Open CASCADE Dev Forum)

Static Blogs (Quaoar, C3D Labs, Geometric Tools)

Incremental Sync

Indexing into Statement-MCP

CLI Pattern

Reference Implementation

Checklist for Adding a New Forum

Common Mistakes

Forum Extraction & Indexing

Overview

Architecture

Output Format

File Organization

Rate Limiting

Platform-Specific Fetchers

Discourse (McNeel Forum, Blender devtalk)

phpBB (FreeCAD Forum)

Drupal (Open CASCADE Dev Forum)

Static Blogs (Quaoar, C3D Labs, Geometric Tools)

Incremental Sync

Indexing into Statement-MCP

CLI Pattern

Reference Implementation

Checklist for Adding a New Forum

Common Mistakes