wechat-mp-rss-debug

name	wechat-mp-rss-debug
version	1.0.0
description	Diagnose why wechat-mp-rss-extractor produces 0 new files - deleted articles (status=1000), slug collisions, and API refresh failures
category	wiki
related	["wechat-mp-rss-extractor","wiki-pipeline"]

we-mp-rss article debugging

When articles appear in RSS feeds but the extractor script produces 0 new files, systematically diagnose using this checklist.

Diagnosis Flow

1. Count new URLs vs inbox

# Get all URLs from inbox
inbox_dir = os.path.expanduser('~/wiki/raw/wechat-inbox/')
inbox_urls = set()
for f in os.listdir(inbox_dir):
    if f.endswith('.md'):
        with open(os.path.join(inbox_dir, f)) as fh:
            for line in fh:
                if line.startswith('source_url:'):
                    inbox_urls.add(line.strip().split('source_url:')[1].strip())
                    break

# Get all URLs from RSS feeds (21 accounts)
# Compare: new_urls = all_rss_urls - inbox_urls

2. Check DB status for each new URL

docker exec we-mp-rss python3 -c "
import sqlite3
conn = sqlite3.connect('/app/data/db.db')
c = conn.cursor()
# NOTE (2026-06-09): schema column is `mp_id` (not `fakeid`), date is `updated_at_millis` (BIGINT ms, not unix seconds)
c.execute('SELECT id, mp_id, status, has_content, length(content), length(content_html), updated_at_millis FROM articles WHERE id LIKE \"%ARTICLE_ID%\" ORDER BY updated_at_millis DESC LIMIT 1')
row = c.fetchone()
print(row)
conn.close()
"

Schema note (2026-06-09 verified): The we-mp-rss DB schema uses mp_id (not the old fakeid field). Timestamps are updated_at_millis (BIGINT milliseconds) and created_at (DATETIME string). Use ORDER BY updated_at_millis DESC for "most recently updated". Quick schema check:

docker exec we-mp-rss python3 -c "
import sqlite3
c = sqlite3.connect('/app/data/db.db').cursor()
print([r[1] for r in c.execute('PRAGMA table_info(articles)').fetchall() if r[1] in ('id','mp_id','status','has_content','content','updated_at_millis')])
"

3. Interpret DB results

status	content	has_content	Meaning
1	>0	1	Valid article - should be written to inbox
1000	DELETED (7 chars)	1	Deleted from WeChat - API returns 404
1000	None	0	Never crawled / removed - API returns 404
1	None	0	Inconsistent state - rare

Tables present (2026-06-09 verified): articles, feeds, users, message_tasks, config_management, access_keys, cascade_nodes, cascade_sync_logs, cascade_task_allocations, filter_rules, message_tasks_logs, tags — note that mp_accounts does NOT exist (use feeds table for account metadata).

Aggregate sanity check (use when extractor returns 0 writes for many URLs):

docker exec we-mp-rss python3 -c "
import sqlite3, time
c = sqlite3.connect('/app/data/db.db').cursor()
now_ms = int(time.time()*1000)
day_ago = now_ms - 24*3600*1000
print('total active:', c.execute('SELECT COUNT(*) FROM articles WHERE status=1 AND has_content=1 AND length(content)>100').fetchone()[0])
print('total deleted (status=1000):', c.execute('SELECT COUNT(*) FROM articles WHERE status=1000').fetchone()[0])
print('last 24h active updated:', c.execute('SELECT COUNT(*) FROM articles WHERE status=1 AND has_content=1 AND length(content)>100 AND updated_at_millis > ?', (day_ago,)).fetchone()[0])
"

If last 24h active updated is 0 even though cron has been firing, the upstream WeChat source is silent — not a script bug. See failure mode #7.

Key insight: status=1000 means the article is filtered out by the API (WHERE status != 1000). The API returns 404 even though the row exists in DB.

4. Check RSS content for valid articles

For articles with status=1, check if RSS content:encoded has content by querying the we-mp-rss RSS endpoint for the specific fakeid.

5. Check for slug collision

If article has content but script skips it, the slug file may already exist with a different URL:

ls ~/wiki/raw/wechat-inbox/ | grep -i "slug-pattern"

Common Failure Modes

1. All new articles deleted (status=1000): The WeChat articles were removed by authors. Script correctly skips them. No action needed.

2. Slug collision: Existing file has same title slug but different URL. The script's os.path.exists(slug + '.md') check prevents overwriting. Fix: rename existing file or update its content.

3. API refresh fails: Articles with empty RSS content trigger POST /refresh, but if status=1000, the refresh task is created but never completes (article not in active set). Script marks as pending_refresh.

4. Script fails to run: ModuleNotFoundError: No module named 'feedparser' - use /usr/bin/python3 (system Python has feedparser + html2text pre-installed).

5. Dry-run produces no output (2026-05-23): python3 scripts/wechat-mp-rss-extractor.py --dry-run silently exits with no output. This means the script hit an import error or silent exit path. Run with full Python to see the error:

   /usr/bin/python3 -c "import sys; sys.path.insert(0, 'scripts'); import wechat_mp_rss_extractor" 2>&1
   

   Also check if Docker container is running: docker ps | grep we-mp-rss

6. Docker DB has no tables (2026-05-23): sqlite3 db.db .tables returns nothing. The database file exists but was never initialized. This happens when the Docker volume was created but the container's init script never ran. Fix: docker exec we-mp-rss python3 -c "from app.database import init_db; init_db()" or restart the container to re-init.

7. All new URLs are status=1000 (upstream silent) — extractor returns 0 writes, this is correct (2026-06-09 verified): When the user runs python3 scripts/wechat-mp-rss-extractor.py --latest=10 and the summary shows rss_empty / api_empty = 8/8 with exists_in_articles: 202 and 0 new writes, the cause is usually that every candidate article in DB is status=1000 (deleted from WeChat). The extractor cannot write what doesn't exist. This is NOT a script bug — it's upstream WeChat state. Diagnose with the aggregate sanity check above: if last 24h active updated is 0, the source is silent. Don't waste cycles restarting the container or re-running the cron — verify state and report honestly:

Quick Diagnostic Script

# Run from ~/wiki
source ~/.wiki-cron.env
/usr/bin/python3 scripts/wechat-mp-rss-extractor.py --latest=10

# Then check:
echo "Inbox count: $(ls raw/wechat-inbox/*.md | wc -l)"
echo "Status log: $(tail -1 cron-status.log)"