Analyze existing Your Project campaigns in place to diagnose prompt, agent, or quality issues — without copying or exporting. Per-scene content classification with agent attribution via debug_info.agent_name. Use when the user asks 'why is the LLM doing X in my campaigns', 'do a regression review across the last N campaigns', 'audit dialog/NPC behavior', 'compare heavy-dialog vs story-mode scenes', or wants prompt-side root-cause diagnosis. Distinct from download-campaign (which exports to disk) and wa-prod-data-query (which analyzes real-user activity).
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Analyze existing Your Project campaigns in place to diagnose prompt, agent, or quality issues — without copying or exporting. Per-scene content classification with agent attribution via debug_info.agent_name. Use when the user asks 'why is the LLM doing X in my campaigns', 'do a regression review across the last N campaigns', 'audit dialog/NPC behavior', 'compare heavy-dialog vs story-mode scenes', or wants prompt-side root-cause diagnosis. Distinct from download-campaign (which exports to disk) and wa-prod-data-query (which analyzes real-user activity).
when_to_use
Use when the user says: analyze the last N campaigns, review dialog quality, audit NPC behavior, regression review, per-scene analysis, agent attribution, why does the LLM X in my campaigns, do all the dialog scenes have Y, find scenes where Z is missing, per-agent breakdown. Do NOT use for: downloading/exporting single campaigns (use download-campaign); real-user activity/retention reports (use wa-prod-data-query); single-bug repros (use repro).
wa-campaign-content-analysis — analyze existing WA campaigns in place
Canonical home for the "look at content already in Firestore and tell me what's happening" workflow. Differs from download-campaign (which exports to disk) and wa-prod-data-query (which analyzes user activity, not scene content). This skill is for prompt / agent / content quality analysis across many existing campaigns without copying or mutating anything.
When to use this skill vs others:
Question class
Skill
"Why does the LLM do X in my campaigns?"
THIS SKILL
"Pull this campaign so I can read it"
download-campaign
"How many real users touched WA last week?"
wa-prod-data-query
"This specific scene has a bug"
repro
"What changed in the prompt last week?"
read git log of $PROJECT_ROOT/prompts/ directly
When to fire this skill
Triggers (any one):
User asks "review the last N campaigns with M+ scenes" or similar cross-campaign content review
User asks "do all my dialog scenes have Y" / "audit NPC behavior" / "is X actually being applied"
User asks "compare heavy-dialog vs story-mode scenes" / per-agent breakdown
User asks "why are NPCs silent" / "why does the PC not speak" / "which agent is generating the response"
After a prompt change — "verify the change actually lands in new scenes"
User names a specific campaign or setting ("the Danerys campaign in Meereen", "the Bran campaign", "the second Visenya run") — see Phase 8 for the user-named-campaign lookup recipe. This is the most common intake shape and the one most likely to mis-route on a literal-name lookup. Do NOT skip to a literal .where("name", "==", ...) query; users often describe by setting/character rather than exact title.
Same venv + auth + clock-skew setup as download-campaign. Do not duplicate it here — read ~/.hermes/skills/download-campaign/SKILL.md Phases 1-3 verbatim and execute them. The firestore connection, WORLDAI_DEV_MODE=true, apply_clock_skew_patch(), and boilerplate are identical.
auth.get_user_by_email("$USER@gmail.com")
Key environment facts:
Project ID: worldarchitecture-ai (with -ture-), NOT worldarchitect-ai
Service account: ~/serviceAccountKey.json
UID for $USER: vnLp2G3m21PJL6kxcuAqmWSOtm73
Subcollection name is story, NOT story_entries (Pitfall from download-campaign)
Phase 2 — Build the candidate campaign list
For "last N campaigns with ≥M scenes":
db = firestore.client()
camps = db.collection("users").document(uid).collection("campaigns").stream()
camp_list = []
for c in camps:
cid = c.id
cd = c.to_dict() or {}
name = cd.get("name") or cd.get("title") or"Untitled"# Aggregation count is the cheap+correct path; .limit(2000).stream() caps at 2000
agg = db.collection("users").document(uid).collection("campaigns").document(cid).collection("story").count().get()
entry_count = int(agg[0][0].value)
if entry_count < MIN_ENTRIES:
continue# Fetch last activity timestamp for sorting
q = (db.collection("users").document(uid).collection("campaigns").document(cid)
.collection("story").order_by("timestamp", direction=firestore.Query.DESCENDING).limit(1).stream())
docs = list(q)
last_ts_raw = Noneif docs:
d = docs[0].to_dict() or {}
last_ts_raw = d.get("timestamp") or d.get("created_at")
camp_list.append({"id": cid, "name": name, "entries": entry_count, "last_ts": norm(last_ts_raw)})
camp_list.sort(key=lambda c: c["last_ts"], reverse=True)
last_n = camp_list[:N]
norm() helper — Firestore timestamps are mixed types in this DB. Some docs store DatetimeWithNanoseconds, others store epoch seconds as int/float. Handle both:
defnorm(v):
if v isNone: return0.0ifhasattr(v, "timestamp"):
try: returnfloat(v.timestamp())
except Exception: passifisinstance(v, (int, float)):
if v > 1e12: returnfloat(v) / 1000.0# millis → secondsreturnfloat(v)
return0.0
Don't sort directly on raw values — TypeError: '<' not supported between instances of 'int' and 'DatetimeWithNanoseconds' fires.
De-duplication note: titles alone are not unique (e.g. "Bran the broken" vs "Bran the broken (ignore directive)" — these are intentional distinct campaigns the user is running, not copies). Only de-duplicate on campaign_id. The download-campaign Pitfall #6 (slug collision on duplicate titles) applies — but for in-place analysis you don't need to worry about it, since you're reading live data.
Phase 3 — Per-scene extraction (the schema insight)
Critical schema facts (verified 2026-07-13 from 2,464 gemini scenes across 10 campaigns):
Field
Meaning
Example values
actor
Who authored this doc
"user", "gemini" (LLM), sometimes "system"
mode
USER INTENT mode — what the player typed
"character", "god", "think", "(unset)"
text
Scene content (markdown narrative + quoted speech)
THE TRAP: mode field is NOT which agent wrote the scene. It records what intent mode the user's input triggered. Across 2,464 scenes in 10 long campaigns, only 3 distinct mode values appeared: character (73%), god (19%), think (7%). Zero scenes have mode=dialog or mode=heavydialog even though 38% of all scenes are written by DialogAgent or HeavyDialogAgent.
Always attribute by debug_info.agent_name, not mode. This is the single most important rule for any cross-campaign analysis.
Schema-zero case (verified 2026-07-26 on Visenya v9, qoQtHsU7DxZnR24VNU9w, 412 scenes): Some campaigns return story docs with debug_info = {} (empty dict — no agent_name, no llm_model, no system_instruction_files) AND full_state_updates = {} (empty dict). Symptom: every per-scene debug_info.agent_name lookup returns None, every per-agent histogram collapses to "unknown", and the typical agent distribution table is meaningless. The narrative quality is fine — the schema fields simply were never written.
Detection before per-agent analysis (cheap, mandatory):
sample = next(camp.collection("story").limit(5).stream()).to_dict() or {}
has_agent_tracking = bool((sample.get("debug_info") or {}).get("agent_name"))
has_state_updates = bool(sample.get("full_state_updates"))
ifnot has_agent_tracking:
print("WARN: empty debug_info — per-agent attribution unavailable")
print("Fallback: analyze raw .txt export at ~/llm_wiki/raw/campaigns/<id8>/")
ifnot has_state_updates:
print("WARN: empty full_state_updates — level / NPC timeline unavailable")
print("Fallback: parse `Status: Lvl N` headers from raw .txt scenes")
When either is empty, do not pretend per-agent findings are real. Report "schema-zero: per-agent attribution unavailable; analyzing raw text export instead." The fallback chain is detailed in references/visenya-v9-raw-text-analyzer.md and the regex patterns in references/personal-scale-challenge-pattern.md.
Phase 4 — Content classifiers
Common per-scene metrics:
Quoted-speech attribution (PC vs NPC)
DIALOG_PATTERNS = [
re.compile(r'"([^"\n]{2,500})"'), # straight double
re.compile(r"'([^'\n]{2,500})'"), # straight single
re.compile(r'"([^"\n]{2,500})"'), # curly double
re.compile(r"'([^'\n]{2,500})'"), # curly single
]
VERB_DIALOG_RE = re.compile(
r"\b([A-Z][a-zA-Z'\-]+(?:\s+[A-Z][a-zA-Z'\-]+)?)\s+"r"(?:said|asked|replied|shouted|whispered|cried|answered|murmured|muttered|stated|declared|"r"exclaimed|gasped|snapped|hissed|sneered|smiled|laughed|grumbled|rejoined|added|continued|noted|"r"remarked|observed|responded|countered|suggested|insisted|protested|begged|pleaded|warned|"r"told|commanded|ordered|demanded|inquired|queried|wondered|breathed|sighed|drawled|babbled)\b"r"[^.!?\n]{0,80}",
flags=re.MULTILINE,
)
NAME_COLON_RE = re.compile(r"^\s*([A-Z][a-zA-Z'\-]+(?:\s+[A-Z][a-zA-Z'\-]+)?)\s*:\s*([^\n]{2,500})",
flags=re.MULTILINE)
FIRST_PERSON_RE = re.compile(r"^(I |I'm |I've |I'll |I'd |My |We |Our |Me |Im |Ive |Ill |Id )",
re.IGNORECASE)
defcount_dialog(text, pc_name, npc_names_set):
"""Return (pc_lines, npc_lines, speakers_list) for a single scene text."""ifnot text:
return0, 0, []
pc_lines = npc_lines = 0
speakers = []
MECHANIC_LABELS = {"Intelligence Check","Wisdom Check","Charisma Check","Persuasion Check",
"Deception Check","Intimidation Check","Performance Check","Insight Check",
"Social HP","Resistance","Objective","Outcome","IMMUNITIES","DISSONANCE",
"IDENTITY","DIVINE BONUSES","LEVEL","DIVINE LEVERAGE","XP",
"Administrative Log","Strategic Note","Session Summary"}
# 1. Quoted speech (dedupe straight+curly overlap)
seen = set()
quoted = []
for pat in DIALOG_PATTERNS:
for m in pat.finditer(text):
line = m.group(1).strip()
key = line[:50]
if key in seen: continue
seen.add(key); quoted.append(line)
for q in quoted:
if FIRST_PERSON_RE.match(q):
pc_lines += 1else:
npc_lines += 1# 2. Name-prefixed dialog ("Name: '...'")for m in NAME_COLON_RE.finditer(text):
sn = m.group(1).strip()
line_text = m.group(2).strip().strip("'\"")
ifnot line_text: continueif sn in MECHANIC_LABELS: continueif pc_name and sn.lower() == pc_name.lower():
pc_lines += 1else:
npc_lines += 1
speakers.append(sn)
# 3. Indirect speech ("Name said that...")for m in VERB_DIALOG_RE.finditer(text):
sn = m.group(1).strip()
if sn in MECHANIC_LABELS: continueif pc_name and sn.lower() == pc_name.lower():
pc_lines += 1else:
npc_lines += 1
speakers.append(sn)
return pc_lines, npc_lines, list(set(speakers))
Limitations of this classifier (verified 2026-07-13):
Misses paraphrased NPC speech (no quotes, no Name said). For PC-silent analysis this is fine (NPC paraphrases still get attributed via verb patterns and NAME_COLON).
First-person quoted lines are heuristic-PC. False positives: third-person narration with self-quoted thought ("My mother always said, 'Don't trust X'"). False negatives: PC speaks in third person ("Bran Stark replies, 'I am the Three-Eyed Raven'"). For long-campaign diagnosis the false-positive rate is acceptable (median PC=0 stays 0 in any case).
Per-game-state pc_name resolution is needed for NAME_COLON/VERB_DIALOG attribution. Pull from users/{uid}/campaigns/{cid}/game_states/current_state.player_character_data.name.
Other useful per-scene metrics
Word count — len(text.split()). Median words/scene per agent reveals scene depth (Combat ~345, HeavyDialog ~436, PlanningAgent ~50).
Quoted-line density — (pc_lines + npc_lines) / words × 100. Low = action-only; high = dialog-heavy.
Scene-bucket histograms — bucket by NPC line count [(0,0),(1,1),(2,3),(4,7),(8,15),(16,31),(32,63),(64+)].
Phase 5 — Per-agent aggregation
Group scenes by debug_info.agent_name (not mode):
from collections import defaultdict
import statistics
by_agent = defaultdict(list)
for s in all_scenes:
by_agent[s["agent"]].append(s)
for agent, sl insorted(by_agent.items(), key=lambda kv: -len(kv[1])):
iflen(sl) < 10: continue
n = len(sl)
print(f"{agent}: scenes={n} med_pc={statistics.median([s['pc_lines'] for s in sl])} "f"med_npc={statistics.median([s['npc_lines'] for s in sl])} "f"%pc_silent={100*sum(1for s in sl if s['pc_lines']==0)/n:.1f}% "f"%two_way={100*sum(1for s in sl if s['pc_lines']>=1and s['npc_lines']>=1)/n:.1f}%")
Typical distribution across long campaigns:
Agent
% of scenes
Notes
HeavyDialogAgent
25-30%
"high-stakes conversations" — should have 2-way dialog; rarely does
GodModeAgent
18-22%
Directive queries; 99%+ PC-silent (player asks, AI answers)
StoryModeAgent
12-16%
Default narrative mode
DialogAgent
10-14%
Active dialog turns (routed via matches_game_state or MODE_DIALOG)
LevelUpAgent
6-10%
Level-up modal — minimal dialog
PlanningAgent
5-8%
Choice generation only — 100% PC-silent, <100 words
CharacterCreationAgent
3-5%
First scene only
CombatAgent
2-4%
Combat — short scenes, NPC monologue for villain banter
FactionManagementAgent
3-4%
Faction minigame
Diagnostic pattern: if a "dialog agent" has % PC-silent > 50%, the prompt is biased toward NPC monologue and needs fixing (see references/pc-silent-prompt-diagnosis.md).
Phase 6 — Verify a previously-shipped fix actually took effect
When the user asks "is the dialog fix working?" / "let me check after deploy" / "is X actually being applied" AFTER a prompt-fix PR has merged, run this verification phase BEFORE assuming the fix worked. The full verification recipe (with regex tightening, pre/post comparison, and 5-cause taxonomy) lives in references/prompt-fix-effectiveness-verification.md. Summary:
Find the merge SHA timestamp from git log of the changed prompt file.
Locate the user's most-recent campaign via users/{uid}/campaigns subcollection (NOT top-level campaigns — that's test fixtures only, per wa-prod-data-query Pitfall #1).
Stream story subcollection and split by merge time. Use the same actor=gemini + mode=<relevant> filter for both sides.
Compute NPC-attributed direct quotes (requires "..." AFTER speech verb within 80 chars — NOT just "Name said" paraphrase, which inflates counts).
Read 2-3 sample scenes verbatim to confirm the classifier isn't fooled by italicized action (*Florent swallows...*) or HUD panels.
If pre/post are statistically indistinguishable, the prompt fix landed but the LLM isn't following. Five causes, in priority order: wrong place in prompt, structural panels crowd, model capacity, wrong bug class fixed, cache miss on prompt update.
Report findings as a pre/post delta + the cause list as options for the user to choose from.
Class-level lesson: prompt-fix-shipped-but-LLM-ignores-it is a distinct bug class from prompt-bug-detected. Verification ≠ detection. The earlier recipe (references/pc-silent-prompt-diagnosis.md) covered detection; this phase covers verification.
Phase 7 — Root-cause diagnosis → prompt files
When per-agent analysis reveals a content-quality regression (e.g. "PC is silent in 70% of dialog scenes"), the cause is almost always in one of three prompt files:
$PROJECT_ROOT/prompts/dialog_system_instruction.md — dialog-specific guidance. Check for NPC-only bias. Search for player character, speak as the PC, PC voice, on behalf of the player — if all zero hits, the prompt never instructs the LLM to put words in the PC's mouth.
$PROJECT_ROOT/prompts/narrative_system_instruction.md (L60-80) and narrative_lite_system_instruction.md (L65-80) — look for "Narrative Authority" / "Player describes / GM describes" blocks. If the GM side reserves NPC reactions without explicitly granting PC dialogue, the LLM defaults to PC-silent.
$PROJECT_ROOT/agents.py — check each agent's REQUIRED_PROMPT_ORDER for whether any PC-voice prompt slot exists. HeavyDialogAgent at L2686-2697 has none.
Don't add backend enforcement. Per root-cause-first skill discipline, fix the prompt layer first. Adding a server-side "PC must speak N times" re-prompt loop is a tail-risk: it doubles token cost on every dialog scene and the LLM will eventually find a way to bypass.
Phase 7 — Persist findings
Three artifacts to produce:
Per-scene JSONL dump at ~/.hermes/<topic>_<date>/all_scenes_by_agent.jsonl — one row per scene with campaign_id, agent, pc_lines, npc_lines, speakers, word count.
Wiki source page at ~/llm_wiki/wiki/sources/<topic>-<date>.md — YAML frontmatter + diagnosis + verbatim sample scenes + prompt citations. Future agents can search this.
Also open a follow-up bead:
cd$HOME/your-project.com && br create "<one-line summary>" \
--type bug --priority 2 --description "<file paths + line numbers + link to wiki page>"
Pitfalls (this list IS the skill — review before running)
Don't de-duplicate by title — "Bran the broken" and "Bran the broken (ignore directive)" are intentionally distinct campaigns the user is running. De-dup only on campaign_id.
Campaign doc name vs title — the campaign-level document uses title (not name) for real-user campaigns. A .where("name", "==", ...) query returns 0 hits even when the campaign exists. See references/campaign-doc-field-naming.md and always read cd.get("title") or cd.get("name"). Verified 2026-07-15, $USER's account.
User-named lookup ≠ literal title — "the Danerys campaign in Meereen" is almost never a literal title match; use the Phase 8 three-step recipe (title pre-filter → scene-level keyword confirmation → opening-scene verification). Don't burn a clarifying menu on the user when the top match is 95% likely correct.
Mixed timestamp types — DatetimeWithNanoseconds vs int epoch vs float epoch-millis. Normalize via the norm() helper before sorting. Sorting raw fails with TypeError: '<' not supported.
/tmp is sandbox-scoped per execute_code call — write artifacts to ~/.hermes/<topic>_<date>/ or ~/llm_wiki/wiki/sources/ instead. Path returned by one execute_code call is gone in the next.
f-string {VAR} in execute_code — the sandbox f-string parser doesn't handle braces inside strings used inside an f-string literal. Write the wiki content via write_file (cleanest) or escape {{/}} everywhere.
Per-campaign PC name resolution — for accurate NAME_COLON/VERB_DIALOG attribution you need player_character_data.name from each campaign's game_states/current_state. Without it, all NAME_COLON lines fall to NPC counter and inflate the npc count. Some campaigns have None for pc_name; fall back to "unknown PC" and accept the inflated NPC count.
First-person PC classifier misses third-person PC speech — fine for PC-silent analysis (median stays 0), bad for quantifying how much PC does speak. If you need precise PC speech counts, sample 10-20 scenes manually.
Test users pollute the data — same as wa-prod-data-query Pitfall 3. For $USER's account this is mostly fine (we filter by uid), but if you ever broaden to multi-uid scans, apply the test-user filter (emails containing test, anon, dev-runner, example.com, jleechantest).
Schema field drift — story-doc schema has evolved over time. Older campaigns may use narrative_text instead of text, created_at instead of timestamp, part instead of id. Normalize field reads: text = d.get("scene_text") or d.get("narrative_text") or d.get("text") or d.get("content"). Same fallback chain for timestamps.
User-intent "mode" vs agent "mode" — these are TWO DIFFERENT FIELDS with the same name. story_doc.mode is user intent. agents.py also uses MODE class constant for agent identity. Don't conflate them.
Don't de-duplicate by title — "Bran the broken" and "Bran the broken (ignore directive)" are intentionally distinct campaigns the user is running. De-dup only on campaign_id.
Schema-zero story docs (debug_info = {}, full_state_updates = {}) — verified 2026-07-26 on Visenya v9. Per-agent attribution falls back to "unknown" for every scene; per-agent histograms and per-agent percentages become meaningless. Detection: run a 5-doc sample probe before the full per-agent aggregation — if (sample.get("debug_info") or {}).get("agent_name") returns None for all 5, the campaign is schema-zero. Fallback: analyze the raw .txt export at ~/llm_wiki/raw/campaigns/<id8>/<title>_<id8>.txt (use download-campaign first). The raw text has ====== SCENE N ====== delimiters and Status: Lvl N ... headers per scene. Patterns documented in references/personal-scale-challenge-pattern.md. Reporting fabricated per-agent percentages (e.g. "HeavyDialogAgent: 25-30%") when the underlying debug_info is empty is dishonest analysis, not a number to summarize.
Sample driver script
See scripts/per_scene_dialog_audit.py (in this skill) for a working end-to-end implementation:
Pulls last N campaigns ≥ M scenes
Streams story subcollection per campaign in ASC timestamp order
Classifies each gemini scene via count_dialog()
Aggregates per-agent
Writes summary JSON + JSONL dump
Related
download-campaign — exports a single campaign (or batch) to disk. Use when the user wants to read campaign content; this skill is for analyzing content in place.
wa-prod-data-query — real-user engagement/retention reports. Wrong domain (user activity, not scene content).
repro — single-bug repro for one specific user-reported scene. This skill is the cross-campaign version.
references/pc-silent-prompt-diagnosis.md — worked example of taking a per-agent breakdown ("PC silent 67%") and tracing it to specific prompt lines.
references/prompt-fix-effectiveness-verification.md — verification recipe for AFTER a prompt fix has shipped: pre/post comparison, regex tightening (NPC-attributed quotes require "..." AFTER speech verb), and 5-cause taxonomy for "PR merged but LLM still not following."
references/personal-scale-challenge-pattern.md — when the LLM auto-escalates to mythic-tier antagonists as the PC scales past the parity band; prompt-only fixes to preserve personal-scale challenge at high tier. Includes the five-prompt-edit recipe (Tier Compression, Consequence-Hiding Heuristic, mythic-NPC personhood, Force-a-Trade, anti-creep on major events) and the regex patterns for measuring personal-scale challenge density from raw text.
references/visenya-v9-raw-text-analyzer.md — pipeline reference for analyzing a campaign whose debug_info is empty; walks through /tmp/analyze_visenya_v9_v2.py style implementation against the download-campaign raw .txt export.
tests/test_agent_attribution.py — verify debug_info.agent_name precedence over mode field.
scripts/test_user_named_campaign_lookup.py — Phase 8 three-step recipe (get_campaign_title, title_prefilter, keyword_scan, confirm_opening_scene) + the campaign-level name vs title field fallback + the "guess and verify, no clarification menu" output contract.
