Download a Your Project campaign from Firestore, write raw archive + wiki source page. Use when the user says download a campaign, pull this campaign, fetch campaign from firestore, ingest campaign N, copy a campaign by id, or wants to walk a specific WA campaign for analysis. Single-campaign or filtered-batch modes. Wraps and supersedes ad-hoc subprocess-based approaches that hit gRPC FD inheritance bugs.
Download a Your Project campaign from Firestore, write raw archive + wiki source page. Use when the user says download a campaign, pull this campaign, fetch campaign from firestore, ingest campaign N, copy a campaign by id, or wants to walk a specific WA campaign for analysis. Single-campaign or filtered-batch modes. Wraps and supersedes ad-hoc subprocess-based approaches that hit gRPC FD inheritance bugs.
when_to_use
Use when the user says: download campaign, fetch this campaign, pull from firestore, ingest campaign, get campaign by id, copy campaign X, batch download recent campaigns, scan last N days, scan for >50 entries, last 2 weeks of campaigns, the campaign wizard walked. Do NOT use for: full wiki page writing (use wiki-ingest), Firestore read only with no download (use firestore_service.get_campaign_by_id directly).
arguments
["campaign_id","filters"]
argument-hint
[campaign_id or 'recent' or 'batch'] [--min-entries N] [--days N] [--skip-existing] [--campaigns-dir <path>]
context
inline
allowed-tools
terminal, file
download-campaign — Your Project Firestore → raw archive + wiki source
Canonical home for the "get a campaign out of Firestore and put it on disk" workflow.
This skill exists because ad-hoc one-liners repeatedly hit three pitfalls:
Subprocess-from-Firebase parents gRPC FD inheritance — download_campaign.py
spawned from a Firebase-initialized process fails with
ev_poll_posix.cc:593 FD from fork parent still in poll list: fd(13, generation: 1)
on every single download. Fix: call firestore_service.get_campaign_by_id() and
document_generator.get_story_text_from_context_enhanced() directly in the same
process. Never use download_campaign.py as a subprocess.
story vs story_entries subcollection — The WA campaign story subcollection
is story. Querying story_entries returns 0 entries for every campaign.
.venv ships without pip and deps — Run ensurepip then install the full
dep chain (firebase-admin, google-cloud-firestore, flask, pydantic, jsonschema,
python-docx, fpdf2). Missing any one causes firestore_service or
document_generator import failure cascades.
Quick usage
cd$HOME/your-project.com
# IMPORTANT: prefix with `env -u MOCK_SERVICES_MODE` to defeat parent-shell# env trap (see Pitfall #13). Required when the parent shell has# MOCK_SERVICES_MODE=true exported (common in hermetic-test sessions).
WORLDAI_DEV_MODE=true \
GOOGLE_APPLICATION_CREDENTIALS=~/serviceAccountKey.json \
env -u MOCK_SERVICES_MODE \
.venv/bin/python ~/.hermes_prod/skills/download-campaign/scripts/download_campaign.py \
--mode one --campaign-id vNU3AAXHd9N7adqWSM2p
# Or batch mode with filters (one user — $USER by default)
.venv/bin/python ~/.hermes_prod/skills/download-campaign/scripts/download_campaign.py \
--mode batch --min-entries 50 --days 14 --skip-existing
# Or all-users mode (every real user, skip test fixtures; $USER included)
.venv/bin/python ~/.hermes_prod/skills/download-campaign/scripts/download_campaign.py \
--mode all-users --min-entries 50 --skip-existing
--mode one --campaign-id <id> — pull a specific campaign (one user, $USER by default).
--mode batch [--min-entries N] [--days N] [--skip-existing] — one user ($USER by default; override with WA_EMAIL env).
--mode all-users [--min-entries N] [--exclude-$USER] [--skip-existing] — every real user across auth.list_users(), filtered by is_test_email(). Use this for the daily cron (wiki-campaign-daily-ingest.sh).
--all-users mode (multi-user batch)
Walks auth.list_users() paginated, filters out test fixtures via
is_test_email() (matches test, anon, dev-runner, example.com,
jleechantest — same tokens as wa-prod-data-query/scripts/query_real_users.py),
then iterates query_candidates() + download_one() per real user. $USER
is included by default; pass --exclude-$USER to skip him.
Each ingested campaign's wiki frontmatter carries user_email: + user_uid:
so multi-user pages are auditable in the jleechanorg/llm-wiki private repo.
Use case: the daily ai.$USER.wiki-campaign-daily-ingest launchd job uses
--mode all-users --min-entries 50 --skip-existing to keep the private
llm-wiki repo in sync with every real WA user's over-50-scene campaigns.
Phases
Phase 1 — Venv bootstrap (one-time per machine)
The WA .venv ships without pip. Run from ~/your-project.com:
All five are required — flask and jsonschema are transitive imports of
firestore_service and document_generator. Missing pydantic breaks
firestore_service.get_campaign_by_id(). Missing python-docx breaks
document_generator.get_story_text_from_context_enhanced(). Missing fpdf2
breaks document_generator import. Install all of them, every time.
Phase 2 — sys.path ordering (every run)
The clock_skew_credentials module lives in $PROJECT_ROOT/, not the project root.
Path order matters — insert mvp_site BEFORE the root:
WORLDAI_DEV_MODE=true is required for any local Firestore query. The
clock-skew validator raises ValueError if WORLDAI_GOOGLE_APPLICATION_CREDENTIALS
is set without WORLDAI_DEV_MODE=true to explicitly acknowledge dev mode.
Phase 3 — Auth + clock skew
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.path.expanduser("~/serviceAccountKey.json")
os.environ["WORLDAI_DEV_MODE"] = "true"from clock_skew_credentials import apply_clock_skew_patch
apply_clock_skew_patch() # adjusts time by -720 secondsimport firebase_admin
from firebase_admin import auth, credentials, firestore
ifnot firebase_admin._apps:
cred = credentials.Certificate(os.path.expanduser("~/serviceAccountKey.json"))
firebase_admin.initialize_app(cred)
user = auth.get_user_by_email("$USER@gmail.com")
uid = user.uid # do not use Firestore user docs for email lookup
Phase 4 — Pull campaign data
import firestore_service
import document_generator
from google.cloud import firestore
campaign_id = "vNU3AAXHd9N7adqWSM2p"
campaign_data, story_context = firestore_service.get_campaign_by_id(uid, campaign_id)
# campaign_data is a Firestore DocumentSnapshot
cd = campaign_data.to_dict() ifhasattr(campaign_data, "to_dict") else campaign_data
title = cd.get("name", cd.get("title", "Untitled"))
# story_context is a list of story entry dicts
story_text = document_generator.get_story_text_from_context_enhanced(
story_context, include_scenes=True
)
# Game state (separate call)
gs = firestore_service.get_campaign_game_state(uid, campaign_id)
gs_data = gs.to_dict() if gs isnotNoneelse {}
Cross-check the helper — see Pitfall #8 below. Some campaigns (e.g.
WlfgzI0ReBrFkmagW3wU "Nocturne bg3 v5 succubus copy") return a
zero-length story_context from firestore_service.get_campaign_by_id
even though the story subcollection has hundreds of entries. The
helper has at least one bug path that drops entries silently. Before
trusting any "0 entries" result, cross-check with a direct
aggregation count and fall back to a direct stream if the helper
count disagrees with the direct count. The driver uses this pattern:
db = firestore.client()
direct_count = int(db.collection("users").document(uid)
.collection("campaigns").document(campaign_id)
.collection("story").count().get()[0][0].value)
if direct_count > len(story_context):
# helper returned truncated/missing — pull directly
docs = list(db.collection("users").document(uid)
.collection("campaigns").document(campaign_id)
.collection("story").stream())
story_context = [d.to_dict() for d in docs]
story_text = document_generator.get_story_text_from_context_enhanced(
story_context, include_scenes=True
)
Phase 5 — Idempotent writes
Two campaigns with the same title (e.g. "Vespera Thul (copy)") must NOT
overwrite each other. Use campaign_id[:8] suffix in BOTH the raw archive
path and the wiki source path:
Idempotency check: skip if wiki_path.exists() AND its size > 500 bytes
(skips blank/partial frontmatter-only stubs).
Phase 5b — Verify the export (mandatory for batch runs)
After a batch run, re-pull each exported campaign from Firestore and
byte-diff the live get_story_text_from_context_enhanced output against
the on-disk raw .txt. Campaigns are live — users keep typing in another
tab while a batch runs, so any candidate that took >1s to export can
have a newer entry by the time the verify pass runs.
Pass criteria:
live_count == on-disk entry_count from frontmatter (no story loss)
len(live_text) within 1% of len(on_disk_text) (allow tiny drift from
new entries; >5% means the export raced with writes and should be
re-pulled)
Output a /tmp/verify_<batch_tag>_report.jsonl with one row per
candidate, then run a repair pass for any row with drift_pct > 1.0
(only those — re-pulling everyone wastes ~3 minutes on a 228-campaign batch).
For the 2026-07-12 batch-50plus run, 28/228 candidates drifted between
batch and verify due to user activity; repair pass re-pulled only those
in ~30s. Total verify+repair wall time: under 4 minutes for 228 candidates.
Phase 6 — Output contract
Each campaign gets:
Raw archive at ~/llm_wiki/raw/campaigns/<id8>/<safe_title>_<id8>.txt
— full story text, scenes included.
Game state at ~/llm_wiki/raw/campaigns/<id8>/<safe_title>_<id8>_game_state.json
— full Firestore game_state doc, JSON serialised with default=str for datetimes.
Wiki source page at ~/llm_wiki/wiki/sources/<slug>-<id8>.md — frontmatter:
Body: the story text, truncated to 100,000 chars per page.
Manifest entry appended to /tmp/campaign_ingest_manifest.jsonl.
Batch mode filters
--min-entries 50 # only campaigns with ≥N story entries
--days 14 # only campaigns active in last N days (by max story timestamp)
--skip-existing # skip if wiki page already exists with content
--campaigns-dir <path> # output raw archive dir (default /tmp/campaign_downloads)
For scanning $USER's full campaign list (1,000+ campaigns in practice),
do NOT use .limit(2000).stream() for entry counts — it caps at 2000 and
gives wrong answers for any campaign with more. Use Firestore aggregation
queries (preferred) or a single .stream() with no limit when count > 2000:
db = firestore.client()
# Preferred: aggregation count (1 read, no document materialization)
agg = (db.collection("users").document(uid).collection("campaigns")
.document(cid).collection("story").count().get())
entry_count = int(agg[0][0].value)
# Fallback for when aggregation isn't enough (need actual entries to sort by# timestamp / author etc.):
docs = list(db.collection("users").document(uid).collection("campaigns")
.document(cid).collection("story").stream())
entry_count = len(docs)
last_scene_ts = max(
(d.to_dict().get("timestamp") or d.to_dict().get("created_at") for d in docs),
default=None,
)
For the 1,080 $USER campaign discovery scan (2026-07-12 batch) the
aggregation path returned correct counts in ~3 minutes total wall time.
Scanning all story subcollections in one process is safe — gRPC FD bug
only fires when download_campaign.py is spawned as a subprocess.
Pitfalls (this list IS the skill — review before running)
Never subprocess download_campaign.py — gRPC FD inheritance
(ev_poll_posix.cc:593 FD from fork parent still in poll list) makes
every download fail. Always inline.
Subcollection is story, not story_entries — story_entries
returns 0 entries. The 2026-05-31 discovery cost a full day of confusion.
WORLDAI_DEV_MODE=true is mandatory — the clock-skew validator
raises ValueError: WORLDAI_GOOGLE_APPLICATION_CREDENTIALS requires WORLDAI_DEV_MODE=true. Set WORLDAI_DEV_MODE=true to explicitly acknowledge development mode. without it.
/private/tmp/types.py collision — if running from /tmp/, a stale
types.py can shadow the stdlib module, causing
ImportError: cannot import name 'GenericAlias'. Fix: rm /private/tmp/types.py.
2000-entry Firestore limit — counting entries with .limit(2000).stream()
shows exactly 2000 for any campaign with 2000+ entries. Use aggregation
queries or no limit for precise counts. For ≥50 detection, limit(2000) is
fine since the threshold is well below the cap.
Slug collision on duplicates — campaigns with the same name (e.g.
"Vespera Thul (copy)") must use campaign_id[:8] in BOTH raw dir name
AND wiki page filename, or the second copy overwrites the first.
.venv dep chain — must install flask pydantic jsonschema python-docx fpdf2
in addition to firebase-admin google-cloud-firestore. The five-app stack
import chain pulls in all of them transitively.
firestore_service.get_campaign_by_id can return an empty story list —
some campaigns silently drop to len(story_context)==0 even though their
story subcollection has hundreds of entries. Discovered 2026-07-12 on
WlfgzI0ReBrFkmagW3wU ("Nocturne bg3 v5 succubus copy", 554 entries on
disk, helper returns 0). Cause: an internal helper bug in
firestore_service.get_campaign_by_id — root-cause not yet isolated, but
the failure mode is consistent (helper returns 0, direct collection stream
returns full count). Fix: always cross-check with
db.collection(...).collection("campaigns").document(cid).collection("story").count().get()
and fall back to a direct .stream() if the direct count > helper count.
Treat any "0 entries" result from the helper as a bug, not as data.
macOS BSD find -newermt @<epoch> is silently broken — use a human-readable
date string instead. GNU find accepts -newermt @1784583090 (epoch
seconds). macOS BSD find does NOT — it silently returns 0 matches and
your batch script will look like "no work was done" when in fact it wrote
a dozen new files. This bites any launchd-managed batch script that uses
find -newermt for change detection. Verified 2026-07-20 on
wiki-campaign-daily-ingest.sh: after the --mode all-users ingest wrote
12 new wiki pages, the shell's find -newermt "@$RUN_START_TIME" returned
0, the script's "Files added/modified: added=0" branch fired, and the
git push was skipped — leaving the 12 new files uncommitted in the
working tree. Fix: store BOTH the epoch AND a formatted date string
(RUN_START_DATE=$(date -r "$RUN_START_TIME" '+%Y-%m-%d %H:%M:%S')) and
use -newermt "$RUN_START_DATE" — works on both BSD and GNU find. The
4 minutes ago relative form also works but is unreliable when the run
wall-clock differs from the script's mtime capture.
slack_post_message | tee swallows the function's return code. When
wrapping slack_post_message (or any helper that can fail) inside an
if guard with | tee, the pipeline's exit code is tee's, not the
helper's — Slack posts silently fail and the script logs "sent". Always
capture $? via $() command substitution:
local out rc
out=$(slack_post_message "$CHAN""$TEXT" 2>&1)
rc=$?
echo"$out" | tee -a "$LOG" >/dev/null
if [[ $rc -eq 0 ]]; thenlog"sent"; elselog"WARN: rc=$rc"; fi
Caught 2026-07-20: bot was posting to #ai-general which it wasn't a
member of, but the script reported "Slack notification sent" anyway.
[[ -z "$X" ]] && X=0 returns non-zero when X is non-empty — the &&
short-circuits to the false branch, and the whole expression exits 1.
Combined with an ERR trap (or set -e), this fires spurious failure
notifications on every successful run. Use explicit assignment:
# WRONG: ERR trap fires on every populated Xeval"[[ -z \"\${$v}\" ]] && $v=0"# RIGHT: plain assignment, always exits 0eval"val=\${$v}"
val=$(printf'%s'"$val" | tr -cd'0-9')
if [[ -z "$val" ]]; then val=0; fieval"$v=$val"
Caught 2026-07-20: the wiki-campaign-daily-ingest.sh ingest-counter
loop was firing notify_error() on every run because the variable
assignment used the wrong pattern.
mcp_agent_mail bot is NOT a member of every channel. The bot
(U0A4G7LDJ4R, app A0A3WSV6BM1) is in #life (C0AMM2B4319) and
#all-$USER-ai (C09GRLXF9GR) but NOT in #ai-general
(C0AJQ5M0A0Y). Verify with conversations.info before picking a
notification channel, or you'll get silent failures. To re-invite the
bot to a new channel, see slack-mcp-mail-bot-reinstall skill §6
(Aside-driven slack.getClient() recipe). For the daily wiki-campaign
cron, #life is the safe default.
MOCK_SERVICES_MODE=true parent shell env silently redirects Firestore to an in-memory mock. Several launchd and
shell sessions export MOCK_SERVICES_MODE=true (it's set in ~/.bashrc / ~/.bash_profile for hermetic-test tooling).
When download_campaign.py runs under such a parent, firestore_service.get_firestore_client() detects the env var
and returns a singleton in-memory client (firestore_service.py L3567-3575). Every Firestore read then returns
"data not found" or empty collections — the script logs MOCK_SERVICES_MODE enabled - creating singleton in-memory Firestore client and exits with RuntimeError: no data for <campaign_id> even though the real campaign has
hundreds of entries. Detection: scan the script's stdout for that exact log line; if present, the env var is
active and you must drop it. Fix: run with env -u MOCK_SERVICES_MODE prefix on the command line
(env -u MOCK_SERVICES_MODE .venv/bin/python ...). Always add this prefix in any download / ingest script
unless you specifically want the mock client. Verified 2026-07-26 on qoQtHsU7DxZnR24VNU9w (Visenya v9):
first run hit RuntimeError: no data for qoQtHsU7DxZnR24VNU9w; with env -u it pulled 824 entries in ~3s.
Game state key fields (for analysis)
player_character_data — PC stats, features, equipment, relationships.
Note: no personality fields (motivation, fear, speech patterns).
npc_data — dict keyed by NPC name; entries have level, role,
relationships, sometimes mbti / alignment (labels only, no deep profile).
~/llm_wiki/tools/batch_campaign_ingest_inline.py — older version of the
batch tool (slug-only paths, no id8 suffix). Superseded by the script
under scripts/download_campaign.py in this skill.
~/llm_wiki/wiki/sources/wa-firestore-campaign-schema.md — schema reference.
references/batch-50plus-2026-07-12.md — session notes from the
228-campaign batch run: scan numbers, verification numbers, driver
scripts, Git workflow, daily-cron opportunity.
references/multi-user-batch-rollout-2026-07-20.md — session notes from
extending the daily cron to all real users (auth.list_users + is_test_email
filter). Includes the find -newermt @<epoch> bug caught during
verification.
Tests
tests/test_slugify.py — slug generation, including id8 suffix.
tests/test_idempotent_path.py — verifies no slug collision for
duplicate titles ("Vespera Thul (copy)" 11× → 11 unique paths).
tests/test_dependencies.py — verifies the .venv dep chain is present.
Run: cd ~/.hermes_prod/skills/download-campaign && python3 -m unittest discover -s tests
